“There is a nice diagram on page 2 that summarizes the contents.”
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
由 22 个子集构成、开源可复现的大规模语言模型训练语料
构建 The Pile:825 GiB 的英文文本语料,由 22 个高质量子集组成,其中多数来自学术或专业来源。评测显示,未调优的 GPT-2 和 GPT-3 在其许多部分(如学术写作)上表现不佳,而用 The Pile 训练的模型在所有组成部分上都明显优于 Raw CC 和 CC-100,下游任务也有提升。作者还对数据中潜在的问题做了探索性分析,并公开了构建代码。
评论者主要把它当作了解大模型训练数据构成的参考:其中包含 HN、arXiv、GitHub 以及 Books3 等来源,常在讨论版权、合理使用和数据质量时被引用。
学习路径
读这篇之前,按顺序读书架上的 5 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出这条路径帮你理解 Pile 出现的背景:预训练语料如何影响模型、缩放定律为何要求更多更多样的数据,以及数据集文档化的做法。
- 1
BERT 确立了在大规模无标注文本上预训练的范式,Pile 的评测就是针对这类预训练。
读读预训练数据(BooksCorpus 与维基百科)和预训练目标,了解当时语料的规模与来源。
- 2
T5 的 C4 是 Pile 对比的主要网页语料,并系统研究了数据清洗与规模。
读读 C4 的构造和过滤规则、数据规模对效果影响的实验。
- 3
缩放定律说明数据量与多样性是继续提升模型的关键,这是 Pile 的直接动机。
读读损失随参数、数据、算力的幂律关系,抓住需要更多数据的结论。
- 4
GPT-3 的训练语料组成(Common Crawl 等)是 Pile 的对照,同时 Pile 用它做评测。
读读训练数据的构成与过滤、数据混合比例的部分,了解 Pile 想改进什么。
- 5
- 6The Pile: An 800GB Dataset of Diverse Text for Language Modeling
给出 22 个子集的构成、处理方法和用 GPT-2、GPT-3 做的评测。
读重点读各子集的来源与权重、去重和过滤,以及跨领域评测的结果。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“This is old and dates from the period surrounding the launch of the 6b model from Eleuther, but have a gander, sheds some light on their sources: The Pile”
“they did some really diligent work in putting together the dataset and documenting it.”
“This includes HN [i] HackerNews 3.90GiB 0.62%”
“My name is on The Pile paper The Pile but I didn't do anything except make the books3 dataset.”
“The answer is to train a general model on a massive dataset, and then to fine-tune it on a smaller dataset”
看全部 20 条 →
“It's mentioned somewhere in here that the language filtering that was used on the source material wasn't perfect”
“It was a part of The Pile, and we included usernames along with the item.”
“This thing is a remarkable abacus for information from The Pile.”
“The paper cited there by contrast, argues for select training sets:”
“Under pre (1984) (and affirmed in subsequent rulings such as aff (2013); Google (2015)), non-commercial, not-for-profit use of copyright media is preemptively fair use.”
“More details in the paper: The Pile”
“In other words, parents aren't trying to teach their children language by reading them to sleep from The Pile[1].”
“The Pile Paper - The Pile - says it was trained (in part) on”
“If you look at The Pile paper[3] for example they explicitly say that much of the material is under copyright and that they are relying on fair use.”
“From The Pile (The Pile: An 800GB Dataset of Diverse Text for Language Modeling)”
“From their paper [0] the reasoning for the book datasets is that they are”
“The Pile dataset for those interested”
“Books3 is a dataset of books derived from a copy of the contents of the Bibliotik private tracker made available by Shawn Presser (Presser, 2020).”
“But, the paper actually describes it as a bizarre mix of curated sources. The Pile”