I want to download a HuggingFace dataset, e.g. uonlp/CulturaX:
from datasets import load_datasetds = load_dataset("uonlp/CulturaX", "en")However, it downloads on one thread at 50 MB/s, while my network is 10 Gbps. Since this dataset is 16 TB, I'd prefer to download it faster so that I don't have to wait for a few days. How can I multithreadedly download a HuggingFace dataset?
