按年被提到
20232023 · 17 次2026
软件工程与 AI 编程›代码生成与程序合成·arXiv:2306.11644 · cs.CL

Textbooks Are All You Need

教科书就是你所需要的一切
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah 等 19 人 · 2023

小模型加教科书级数据,写代码胜过大模型

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 22 人首次 2023-06-21 · 最近 2026-08-15
22不同的人
11活跃月份
23讨论串
428被引(OpenAlex)

介绍代码大模型 phi-1:1.3B 参数的 Transformer,用 8 块 A100 训练 4 天,数据为 6B token 的教科书质量网络数据加 1B token 由 GPT-3.5 生成的教科书和习题。它在 HumanEval 上 pass@1 达 50.6%,在 MBPP 上达 55.5%,说明高质量数据可让小模型在代码任务上表现出色,并出现意外的涌现能力。

HN 为什么提它

评论者把它当作数据质量胜过数据规模和算力的代表例证,也提到它推广了用精心构造的合成数据训练模型的做法。

实证研究难度 2 / 5约 35 分钟有时效性
读之前最好知道
1Transformer 架构
2大语言模型预训练与微调
3代码生成评测(HumanEval、pass@1)
4合成数据(synthetic data)

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
强烈推荐lhl · 2023-07

“I highly recommend reading the MSR phi-1 paper:”

强烈推荐 MSR 的 phi-1 论文,1.3B 模型用很少 token 就有出色代码表现
LLaMA2 Chat 70B outperformed ChatGPT原评论 ↗
推荐bluecoconut · 2023-06

“the results seem to suggest that careful curation and curriculums of training data can significantly improve model capabilities (when training domain specific, smaller models).”

推荐它,结果表明精心筛选和编排训练数据能显著提升小型领域模型
The bigger-is-better approach to AI is running out of road原评论 ↗
推荐joshhart · 2023-07

“There are many good papers on the importance of data quality, Textbooks Are All You Need is one that comes to mind”

举它为数据质量重要性的好论文之一
Llama 2 – Meta AI原评论 ↗
推荐shmed · 2026-07

“The accompanying paper Textbooks Are All You Need has been pretty instrumental in pushing the idea of highly curated synthetic dataset to train model.”

称这篇论文对推广精心筛选的合成数据集训练模型起了重要作用
MAI-Cyber 1原评论 ↗
引用astrange · 2023-06

“Less data can be better if the data is good: Textbooks Are All You Need”

引用它说明数据好的话,更少的数据反而更好
How RLHF Works原评论 ↗
引用PeterStuer · 2023-06

“The phi-1 paper suggests small models trained on higher quality data hold promise as well.”

引用 phi-1:在更高质量数据上训练的小模型也有前景
The False Promise of Open-sourced LLMs context length原评论 ↗
看全部 20 条 →
引用throwaway4aday · 2023-06

“The other paper[2] demonstrates using a combination of curated web data and machine generated textbooks and exercises (using GPT-3.5) and they produced a small model (1.3B) that outperforms several larger models on programming tasks.”

引用它:用筛选网页数据加 GPT-3.5 生成的教科书训练的 1.3B 模型在编程任务上胜过更大的模型
The Darwinian argument for worrying about AI原评论 ↗
引用petesergeant · 2023-07

“My recent reading[0, 1] suggests AI training with AI-generated content provides excellent results”

引作证据:用 AI 生成内容训练 AI 效果很好
Mechanical Turk workers are using AI to automate being human原评论 ↗
引用espadrine · 2023-07

“like Phi-1[1] (a 1.3B code model outperforming 15B+ models)”

举 Phi-1 为例:1.3B 代码模型胜过 15B 以上的模型
Which GPU(s) to Get for Deep Learning原评论 ↗
引用csjh · 2023-08

“phi-1[0] is only 1.3 billion parameters and performs very well in coding tasks”

举 phi-1 为例:仅 13 亿参数,编码任务表现很好
StableCode原评论 ↗
引用sbierwagen · 2023-09

“phi-1 supposedly does 50.6 on HumanEval with 1.3B parameters. (Python only) Textbooks Are All You Need”

引用 phi-1 在 HumanEval 上 50.6 的成绩,仅 1.3B 参数且只针对 Python
Releasing Persimmon-8B原评论 ↗
引用theptip · 2023-09

“Sounds vaguely like the paper “textbooks are all you need”? Though they are not explicitly trying to remove the copyright taint.”

联想到这篇论文,但指出它并非要去除版权污染
Optimizing LLMs from a Dataset Perspective原评论 ↗
引用ironborn123 · 2023-10

“Wasnt there a paper a few months back, Textbooks are all you need. yes found it Textbooks Are All You Need”

回忆起几个月前这篇教科书质量训练数据的论文,并找到了它
Chatbot Hallucinations Are Poisoning Web Search原评论 ↗
引用pk-protect-ai · 2023-10

“Exactly. And that is why such texts must be excluded.”

认同其思路:因此这类文本必须被排除
LoRA Fine-Tuning Efficiently Undoes Safety Training from Llama 2-Chat 70B原评论 ↗
引用PrayagBhakar · 2023-10

“We know LLMs perform way better when you use high quality input data [0].”

引作证据:高质量输入数据让 LLM 表现好得多
Anna's Archive – LLM Training Data from Shadow Libraries原评论 ↗
引用gardnr · 2023-10

“A good example of this is the "Textbooks Are All You Need" [5] paper that propsed pre-training in an incremental way, similar to how humans learn, produces novel results.”

举它为例:像人类学习那样渐进式预训练,得到新颖结果
Pretraining on the Test Set Is All You Need原评论 ↗
引用killerstorm · 2023-11

“E.g. check "Textbooks Are All You Need" paper: Textbooks Are All You Need A small model trained on high-quality dataset can beat much bigger models at code generation.”

引用它:在高质量数据上训练的小模型在代码生成上能胜过大得多的模型
OpenAI Announces Leadership Transition原评论 ↗
引用emporas · 2024-01

“As the experiments on PHI-1 and PHI-2 from microsoft show, training data make a difference.”

引用 Phi-1 和 Phi-2 的实验,说明训练数据确实有影响
Chess-GPT's Internal World Model原评论 ↗
引用pclmulqdq · 2024-07

“presenting a very limited set of information to an LLM gets a decent result”

引它说明只给 LLM 很有限的精选信息也能得到不错的结果
Judge dismisses DMCA copyright claim in GitHub Copilot suit原评论 ↗
引用anankaie · 2024-07

“data seems to be much more important than compute”

引它说明数据似乎比算力重要得多
Apple says it uses no Nvidia GPUs to train its AI models原评论 ↗