引用dwagnerkc · 2024-02
“That post also very helpfully links to another paper they published alongside the OLMo paper just on the dataset.”
指出该帖链接了与 OLMo 论文同时发布、专讲数据集的另一篇论文
发布并详述三万亿词元英文语料 Dolma 及其数据整理工具
最先进语言模型使用的预训练语料很少被公开讨论,这使得研究训练数据如何影响模型能力和局限变得困难。作者整理并发布 Dolma,一个三万亿词元的英文语料库,来源包括网页、科学论文、代码、公版图书、社交媒体和百科材料,并详细记录其设计原则、构建细节和内容概览。文中给出在 Dolma 中间状态上的分析与实验,分享关于数据整理实践的经验,并开源大规模数据整理工具包以便复现和后续研究。
评论者把它称为与 OLMo 论文同时发布的专门介绍数据集的论文,并引用其流水线用 fastText 毒性分类器清洗语料,也用它说明开放预训练语料的构成。
“That post also very helpfully links to another paper they published alongside the OLMo paper just on the dataset.”
“MIT, the Steel Factory, Cal, UDub and what, 37 authors pushed a preprint and no one heard about it?”
“e.g. the fastText toxicity classifiers in the Dolma pipeline [2]”
“None, since 'legal' for AI training is not yet defined, but Olma is trained on the Dolma 3 dataset, which is”