按年被提到
20242024 · 1 次2026
大模型与语言模型›模型表达力与计算理论·arXiv:2407.12034 · cs.CL

Understanding Transformers via N-gram Statistics

Timothy Nguyen · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-12-18 · 最近 2025-05-17
2不同的人
2活跃月份
2讨论串
4被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Transformer based large-language models (LLMs) display extreme proficiency with language yet a precise understanding of how they work remains elusive. One way of demystifying transformer predictions would be to describe how they depend on their context in terms of simple template functions. This paper takes a first step in this direction by considering families of functions (i.e. rules) formed out of simple N-gram based statistics of the training data. By studying how well these rulesets approximate transformer predictions, we obtain a variety of novel discoveries: a simple method to detect overfitting during training without using a holdout set, a quantitative measure of how transformers progress from learning simple to more complex statistical rules over the course of training, a model-variance criterion governing when transformer predictions tend to be described by N-gram rules, and insights into how well transformers can be approximated by N-gram rulesets in the limit where these rulesets become increasingly complex. In this latter direction, we find that for 79% and 68% of LLM next-token distributions on TinyStories and Wikipedia, respectively, their top-1 predictions agree with those provided by our N-gram rulesets.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论cma · 2024-12

“GPT4o also has RLHF tuning on top of it though. For raw LLMs getting kind of tripped up by being something like interpolative n-gram estimators this is some recent work that does put them closer to markov chains in some ways, especially early on in training but not fully https://arxiv.org/abs/2407.12034 "In this latter direction, we find that for 79% an…”

Google, the search engine that's forgotten how to search原评论 ↗
被提交为 story · 1 次