The Pile is an 825 GiB English-language text corpus assembled by EleutherAI specifically for training large language models. Rather than a single web scrape, it combines 22 distinct sub-datasets chosen to maximize diversity, spanning academic writing, books, web text, code, dialogue, legal, and medical sources. Many components come from curated or professional origins in order to raise overall quality relative to generic crawled text.
What’s in it
Notable subsets include PubMed and arXiv papers, GitHub code, Wikipedia, Stack Exchange, USPTO patents, subtitles, and web crawls (Pile-CC), among others, each documented with its provenance and mixing weight. This deliberate mixture aims to expose models to varied domains, registers, and reasoning styles, and the paper reports per-subset statistics so practitioners can reproduce or reweight the blend. It is distributed as compressed JSON-lines shards and is used both as a full pretraining set and as a source of held-out, per-domain evaluation splits.
Tasks, license & access
The Pile is used to pretrain and benchmark autoregressive language models, including EleutherAI’s own GPT-Neo and GPT-J, and it remains a common perplexity-evaluation corpus. It was released publicly in late 2020; licensing is per-component, since each sub-dataset keeps its original terms, and some copyrighted portions (such as the Books3 subset) were later removed from official distributions over legal concerns.
- Modality: plain English text
- Scale: 825 GiB across 22 sub-datasets
- Tasks: LLM pretraining, language-model benchmarking
- License: mixed/per-component; some subsets withdrawn
- Creator: EleutherAI
As one of the first openly available, well-documented pretraining corpora, The Pile helped make large-language-model research reproducible outside big industry labs.