按年被提到
20232023 · 20 次2026
大模型与语言模型›大模型预训练与缩放·arXiv:2305.07759 · cs.CL

TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

TinyStories:语言模型能小到什么程度,仍能说出连贯的英语?
Ronen Eldan, Yuanzhi Li · 2023

用合成幼儿故事训练极小模型,仍能写出连贯英语

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 23 人首次 2023-05-16 · 最近 2026-09-29
23不同的人
15活跃月份
30讨论串
177被引(OpenAlex)

介绍 TinyStories:由 GPT-3.5 和 GPT-4 生成、只用 3 到 4 岁孩子通常能懂的词汇写成的合成短故事数据集。用它训练的模型远小于主流模型(参数不到 1000 万,甚至只有一个 Transformer 块),仍能写出流畅、前后一致、语法近乎完美、带有一定推理能力的多段故事。论文还提出用 GPT-4 像老师给学生作文打分那样,从语法、创造性、一致性等维度评价语言模型。

HN 为什么提它

评论者用它说明:不要求学会全部人类知识时,小模型加精心构造的合成数据就能说连贯英语,数据质量能大幅降低规模门槛,并借鉴它用 LLM 做评分的评测方式。

实证研究难度 2 / 5约 40 分钟经得起时间
读之前最好知道
1语言模型与 Transformer 基础
2合成数据(synthetic data)
3模型规模与数据量的关系
4LLM 作为评审(LLM-as-judge)

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐blackkettle · 2023-07

“I think there may be some applications in this limited space that are worth looking into.”

认为在这类受限范围内有些应用值得研究
How to scale LLMs better with an alternative to transformers原评论 ↗
推荐dmezzetti · 2024-01

“Couple other interesting papers from 2023 to add, specifically for small language models.”

补充 2023 年几篇小语言模型的论文,称好而精炼的训练数据能让小模型表现更好
Ten Noteworthy AI Research Papers of 2023原评论 ↗
推荐robrenaud · 2024-01

“The TinyStories[1] paper has an interesting solution for how to evaluate stories.”

指出这篇论文评估故事的方法很有意思:让 GPT-4 给故事打分
Launch HN: Talc AI (YC S23) – Test Sets for AI原评论 ↗
推荐azath92 · 2025-12

“check out the tiny stories dataset that limits vocab to a certain age but keeps core reasoning inherent in even simple language [LINK] TinyStories”

推荐看 TinyStories 数据集:词汇限定到某个年龄,仍保留简单语言中的核心推理
LLM from scratch, part 28 – training a base model from scratch on an RTX 3090原评论 ↗
批评CaptainOfCoit · 2024-05

“Hardly high quality "story", and an LLM training on data like that won't have high quality output no matter how much you train it.”

“Imprecise” language models are smaller, speedier, and nearly as accurate原评论 ↗
引用visarga · 2023-05

“A recent paper (TinyStories) shows you can train a LM entirely with synthetic data and even evaluate it with another LLM.”

引用它说明语言模型可以完全用合成数据训练,并用另一个 LLM 评估
Apple bans ChatGPT use by employees, report says原评论 ↗
看全部 20 条 →
引用og_kalu · 2023-06

“Depending on what exactly you're trying to teach (perfect grammar, paragraphs of coherent text, basic reasoning), much less data is needed. TinyStories”

引用它说明取决于要教什么,需要的数据可以少得多
Modern language models refute Chomsky’s approach to language原评论 ↗
引用lhl · 2023-06

“It looks like the 28M parameter 8 layer model had a 9/10 on grammar and consistency.”

指出 28M 参数、8 层的模型在语法和一致性上拿到了 9/10
LLM Powered Autonomous Agents原评论 ↗
引用photonthug · 2023-06

“doesn't your logic here advocate throwing away every "lower-bound" type of result in math, suggest that it's boring/trivial to answer the smallest LLM that speaks english, etc?”

拿它反驳对方逻辑:“能说英语的最小 LLM 有多小”也是个值得回答的下界问题
Integrated Information Theory原评论 ↗
引用bawolff · 2023-06

“The former is an interesting question.”

认为“语言模型能小到什么程度仍能说英语”是个有意思的问题
Integrated Information Theory原评论 ↗
引用petesergeant · 2023-07

“My recent reading[0, 1] suggests AI training with AI-generated content provides excellent results”

引用它说明用 AI 生成的内容训练 AI 可以得到很好的结果
Mechanical Turk workers are using AI to automate being human原评论 ↗
引用patapong · 2023-10

“It seems like small models are emerging that are a bit more specific but can be very small, and thus have much lower latency. For example: TinyStories”

引用它说明正在出现更专门却可以很小的模型,延迟低得多
Disney Packed Big Emotion into a Little Robot原评论 ↗
引用ronsor · 2023-10

“The TinyStories paper[0] explores some of the potential of”

引用它探讨小模型配上高质量数据仍能产出连贯英语的潜力
Consortium launched to build the largest open LLM原评论 ↗
引用ckcheng · 2024-01

“Or did you mean Ronen Eldan and Yuanzhi Li's”

指出 TinyStories 的作者是 Eldan 和 Li,并贴出论文
Stuff we figured out about AI in 2023原评论 ↗
引用OmegaPoint · 2024-05

“Synthetic data are the answers. For example see Tiny Stories dataset.”

称合成数据就是答案,以 TinyStories 数据集为例
“Imprecise” language models are smaller, speedier, and nearly as accurate原评论 ↗
引用hagen_dogs · 2024-12

“The essential point (of the TinyStories paper), "if we train a model on text meant for 3-4 year olds, since that's much simpler shouldn't we need fewer parameters?" Which is correct.”

概括论文要点:用适合 3-4 岁孩子的简单文本训练,参数应该可以更少,这是对的
Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning原评论 ↗
引用exe34 · 2025-01

“This is similar to TinyStories, except here they're training something even better than the original model for a fraction of the price.”

说这与 TinyStories 类似,只是这里花很少的钱训练出比原模型更好的东西
OpenAI says it has evidence DeepSeek used its model to train competitor原评论 ↗
引用rakejake · 2025-03

“The only one I was able to find was MS's TinyStories [0], but that paper didn't delve much into how they trained the models”

说只找到微软的 TinyStories 这一篇,但论文没怎么讲模型怎么训练
Low responsiveness of ML models to critical or deteriorating health conditions原评论 ↗
引用ssivark · 2025-11

“The problem with HMMs is that the sequence model (Markov transition matrix) accounts for much less context than even Tiny LLMs.”

拿小型 LLM 作参照:HMM 的序列模型能利用的上下文比它们还少得多
Ask HN: How are Markov chains so different from tiny LLMs?原评论 ↗
引用gregsadetsky · 2026-09

“Today, I (Fable) ported tinystories [3], a very small language model to this exact same chip.”

说自己把 TinyStories 这个很小的语言模型移植到了同一块芯片上
WebFPGA (2019)原评论 ↗