“I highly recommend reading the MSR phi-1 paper:”
Textbooks Are All You Need
小模型加教科书级数据,写代码胜过大模型
介绍代码大模型 phi-1:1.3B 参数的 Transformer,用 8 块 A100 训练 4 天,数据为 6B token 的教科书质量网络数据加 1B token 由 GPT-3.5 生成的教科书和习题。它在 HumanEval 上 pass@1 达 50.6%,在 MBPP 上达 55.5%,说明高质量数据可让小模型在代码任务上表现出色,并出现意外的涌现能力。
评论者把它当作数据质量胜过数据规模和算力的代表例证,也提到它推广了用精心构造的合成数据训练模型的做法。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“the results seem to suggest that careful curation and curriculums of training data can significantly improve model capabilities (when training domain specific, smaller models).”
“There are many good papers on the importance of data quality, Textbooks Are All You Need is one that comes to mind”
“The accompanying paper Textbooks Are All You Need has been pretty instrumental in pushing the idea of highly curated synthetic dataset to train model.”
“Less data can be better if the data is good: Textbooks Are All You Need”
“The phi-1 paper suggests small models trained on higher quality data hold promise as well.”
看全部 20 条 →
“The other paper[2] demonstrates using a combination of curated web data and machine generated textbooks and exercises (using GPT-3.5) and they produced a small model (1.3B) that outperforms several larger models on programming tasks.”
“My recent reading[0, 1] suggests AI training with AI-generated content provides excellent results”
“like Phi-1[1] (a 1.3B code model outperforming 15B+ models)”
“phi-1[0] is only 1.3 billion parameters and performs very well in coding tasks”
“phi-1 supposedly does 50.6 on HumanEval with 1.3B parameters. (Python only) Textbooks Are All You Need”
“Sounds vaguely like the paper “textbooks are all you need”? Though they are not explicitly trying to remove the copyright taint.”
“Wasnt there a paper a few months back, Textbooks are all you need. yes found it Textbooks Are All You Need”
“Exactly. And that is why such texts must be excluded.”
“We know LLMs perform way better when you use high quality input data [0].”
“A good example of this is the "Textbooks Are All You Need" [5] paper that propsed pre-training in an incremental way, similar to how humans learn, produces novel results.”
“E.g. check "Textbooks Are All You Need" paper: Textbooks Are All You Need A small model trained on high-quality dataset can beat much bigger models at code generation.”
“As the experiments on PHI-1 and PHI-2 from microsoft show, training data make a difference.”
“presenting a very limited set of information to an LLM gets a decent result”
“data seems to be much more important than compute”