Language Modeling

Filter

Modalities

Multimodal Text Image Tabular Video Audio Sensor / Time-series 3D / Point Cloud LiDAR / Radar

Domains

General Robotics / Physical AI Healthcare Geospatial & Earth Observation Finance Automotive & Mobility Legal

Licenses

Apache-2.0 CC-BY MIT Custom / Research-only CC-BY-NC CC-BY-NC-SA CC-BY-SA CC0 Other ODC-BY Mixed GPL-3.0 AGPL-3.0 BSD-3-Clause CC-BY-NC-ND OpenRAIL ODbL BSD MPL-2.0 AFL-3.0 LGPL-3.0 Artistic-2.0 BSD-2-Clause Llama-3 License ISC Unlicense

Access levels

Open 2922 Gated 293

Sizes

10K–100K 748 1K–10K 598 100K–1M 595 <1K 439 1M–10M 329 10M–100M 169 100M–1B 93 >1B 74 >10M 3

Formats

Parquet 1501 Images (folder) 369 JSON 248 CSV 173 Text (raw) 66 WebDataset 62 Audio (folder) 54 Arrow 15 Audio (wav/mp3) 3 Point Cloud (LAS/PLY) 2 Video (mp4) 1 rosbag 1 COCO JSON 1

Languages

English 1233 Multilingual 575 Chinese 36 Arabic 34 Japanese 25 Russian 18 Vietnamese 17 French 9 Portuguese 8 Spanish 7 Persian 7 Turkish 6 German 5 Korean 5 Italian 4 Ukrainian 3 Thai 3 Bengali 2 Romanian 2 Hindi 2 Hebrew 2 Tamil 2 Dutch 2 Swedish 1 Indonesian 1 Czech 1 Greek 1

1 result

Text

The Pile

825 GiB of diverse text from 22 curated sources.

>10M·Mixed·Parquet

DataTrain.AI

AI Data Pipelines for Training, Processing, and Multimodal Workflows

Browse

Datasets
Companies
Tools

Resources

Blog
Submit a listing

Language Modeling

The Pile

Archives

Categories