fineweb_deduplicated

TL;DR Fineweb is a popular and high quality open dataset. This dataset is a deduplicated version of Fineweb – removing rows with duplicate text, collecting counts. Motivation Fineweb is an open text dataset intended for training language models. It’s one of the highest quality and most popular open datasets available. It has been produced by a reputable AI lab – HuggingFace and has been downloaded tens of thousands of times. Fineweb dataset is 93.4 TB and has 15T… See the full description on the dataset page: deduplicated.

Source: Hugging Face Hub (Salesforce/fineweb_deduplicated). Metadata imported from the dataset’s Hub tags.