推荐IshanMi · 2024-03
“A more recent and even larger dataset would be the Falcon-RefinedWeb dataset [2].”
补充一个更新更大的数据集 Falcon-RefinedWeb
仅用过滤去重后的网页数据训练出强于精选语料的 LLM
论文挑战了大模型预训练必须混合精选语料的看法,表明经过严格过滤和去重的纯网页数据足以训练出强大模型,甚至显著优于在 The Pile 上训练的最先进模型。作者从 CommonCrawl 中提取出五万亿 token 的高质量数据,并公开了其中 6000 亿 token 的子集,以及 1.3B 和 7.5B 参数的模型。
HN 上的人把它当作大型开放网页训练数据集的例子,用来说明 LLM 数据如何构建,也常提到 HN 是其中被排除的少数网站之一。
“A more recent and even larger dataset would be the Falcon-RefinedWeb dataset [2].”
“For example, the RefinedWeb paper lists HN as one of only 12 websites that were excluded.”
“If you take all public web text it is just something like 40 trillion characters”
“For example, the RefinedWeb paper lists HN as one of only 12 websites that were excluded.”