“I think there may be some applications in this limited space that are worth looking into.”
TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
用合成幼儿故事训练极小模型,仍能写出连贯英语
介绍 TinyStories:由 GPT-3.5 和 GPT-4 生成、只用 3 到 4 岁孩子通常能懂的词汇写成的合成短故事数据集。用它训练的模型远小于主流模型(参数不到 1000 万,甚至只有一个 Transformer 块),仍能写出流畅、前后一致、语法近乎完美、带有一定推理能力的多段故事。论文还提出用 GPT-4 像老师给学生作文打分那样,从语法、创造性、一致性等维度评价语言模型。
评论者用它说明:不要求学会全部人类知识时,小模型加精心构造的合成数据就能说连贯英语,数据质量能大幅降低规模门槛,并借鉴它用 LLM 做评分的评测方式。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Couple other interesting papers from 2023 to add, specifically for small language models.”
“The TinyStories[1] paper has an interesting solution for how to evaluate stories.”
“check out the tiny stories dataset that limits vocab to a certain age but keeps core reasoning inherent in even simple language [LINK] TinyStories”
“Hardly high quality "story", and an LLM training on data like that won't have high quality output no matter how much you train it.”
“A recent paper (TinyStories) shows you can train a LM entirely with synthetic data and even evaluate it with another LLM.”
看全部 20 条 →
“Depending on what exactly you're trying to teach (perfect grammar, paragraphs of coherent text, basic reasoning), much less data is needed. TinyStories”
“It looks like the 28M parameter 8 layer model had a 9/10 on grammar and consistency.”
“doesn't your logic here advocate throwing away every "lower-bound" type of result in math, suggest that it's boring/trivial to answer the smallest LLM that speaks english, etc?”
“The former is an interesting question.”
“My recent reading[0, 1] suggests AI training with AI-generated content provides excellent results”
“It seems like small models are emerging that are a bit more specific but can be very small, and thus have much lower latency. For example: TinyStories”
“The TinyStories paper[0] explores some of the potential of”
“Or did you mean Ronen Eldan and Yuanzhi Li's”
“Synthetic data are the answers. For example see Tiny Stories dataset.”
“The essential point (of the TinyStories paper), "if we train a model on text meant for 3-4 year olds, since that's much simpler shouldn't we need fewer parameters?" Which is correct.”
“This is similar to TinyStories, except here they're training something even better than the original model for a fraction of the price.”
“The only one I was able to find was MS's TinyStories [0], but that paper didn't delve much into how they trained the models”
“The problem with HMMs is that the sequence model (Markov transition matrix) accounts for much less context than even Tiny LLMs.”
“Today, I (Fable) ported tinystories [3], a very small language model to this exact same chip.”