按年被提到
20232026 · 8 次2026
大模型与语言模型›量化、解码与分词·arXiv:2309.10668 · cs.LG

Language Modeling Is Compression

语言建模即压缩
Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, Joel Veness · 2023

从压缩视角看语言模型,Chinchilla 压缩图像胜过 PNG

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 26 人首次 2023-09-20 · 最近 2026-11-02
26不同的人
18活跃月份
26讨论串
93被引(OpenAlex)

论文指出预测模型与无损压缩器可以互相转换,主张用压缩的视角看待预测问题,并评估大型基础模型的压缩能力。例如主要用文本训练的 Chinchilla 70B 能把 ImageNet 图像块压到原大小的 43.4%、LibriSpeech 音频压到 16.4%,优于 PNG(58.5%)和 FLAC(30.3%);压缩视角还对缩放定律、分词和上下文学习提供新的见解,任何压缩器(如 gzip)也可用来构造条件生成模型。

HN 为什么提它

评论者把它当作「语言建模即压缩」的出处来引用,用来论证 LLM 本质上是无损压缩,以及仅用文本训练的 Transformer 也能压缩图像。

实证研究难度 3 / 5约 40 分钟经得起时间
读之前最好知道
1信息论(熵与算术编码)
2语言模型与 Transformer
3无损压缩基础
4缩放定律(scaling laws)

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐TheAlchemist · 2024-07

“This is one paper you could read”

推荐读这篇论文,它把语言模型用作数据压缩器,对象是文本而非影片
Ask HN: How efficient are MLs at storing data?原评论 ↗
引用m00x · 2023-09

“It goes along the same lines as the recent paper from Deepmind stating that DL (language modeling in their case) is compression.”

指出 DeepMind 这篇论文也主张深度学习(语言建模)就是压缩
Show HN: 78% MNIST accuracy using GZIP in under 10 lines of code原评论 ↗
引用intended · 2023-10

“I recall there was a paper with a relevant title recently… Language Modeling Is Compression”

回忆起一篇相关论文,把 LLM 预测看作无损压缩
OpenAI is too cheap to beat原评论 ↗
引用atorodius · 2023-12

“Many papers on this, recent one”

举它为用下一个 token 预测做压缩的近期例子
Adventures with compression原评论 ↗
引用jabowery · 2024-01

“The increasing recognition that "Language Modeling Is Compression" Language Modeling Is Compression has not yet been accompanied by recognition”

引用它指出“语言建模即压缩”的认识尚未被充分重视
Getting Lossless Compression Adopted for Rigorous LLM Benchmarking原评论 ↗
引用haltIncomplete · 2024-03

“All we’re doing is engineering new data compression and retrieval techniques: Language Modeling Is Compression”

拿它说明语言模型是强大的通用压缩器,所做的只是新的压缩与检索技术
LLMs use a surprisingly simple mechanism to retrieve some stored knowledge原评论 ↗
看全部 20 条 →
引用gwern · 2024-04

“He just needed to use LLMs instead: Language Modeling Is Compression”

指出用 LLM 做压缩可以替代手工构建的系统
EURISKO Lives原评论 ↗
引用bugglebeetle · 2024-08

“Oh well, you should tell the folks at DeepMind and Meta about these objective facts”

用它反驳上文观点:语言建模就等于压缩
Procreate's anti-AI pledge attracts praise from digital creatives原评论 ↗
引用bangertho · 2025-05

“as we continue to compress from the model all the non-essential syntax sugar of modern software.”

把语言模型即压缩器的核心想法用来类比,模型会压掉现代软件里非必要的语法糖
The copilot delusion原评论 ↗
引用braaaahp · 2025-05

“Edit: Language Modeling Is Compression”

补充链接,指向语言模型即压缩器的论文
Lossless video compression using Bloom filters原评论 ↗
引用RadEng00 · 2025-06

“I look forward to continuing my work in the hardware world to further compress and reduce the useless state of past systems of though we copy paste around to serve developers, to reduce context to sort through, and improve model quality: Language Modeling Is Compression”

引用它支持用压缩思路精简历史系统的冗余状态、提升模型质量
The Illusion of Thinking: Strengths and limitations of reasoning models [pdf]原评论 ↗
引用Mathnerd314 · 2025-06

“are essentially compression algorithms?”

想起 LLM 本质上是压缩算法,并联系到微调
Fine-tuning LLMs is a waste of time原评论 ↗
引用b112 · 2025-07

“Not to mention, Language Modeling is Compression Language Modeling Is Compression”

引用它作为“语言建模即压缩”的论据
How large are large language models?原评论 ↗
引用disruptbro · 2025-07

“Language modeling is compression, whittle down graph to reduce duplication and data with little relationship: Language Modeling Is Compression”

借“语言建模即压缩”的观点来说明精简图以减少重复数据
Quantitative AI progress needs accurate and transparent evaluation原评论 ↗
引用tekbruh9000 · 2025-10

“LLMs are a form of compression [1].”

引用它说明 LLM 是一种压缩
Andrej Karpathy – It will take a decade to work through the issues with agents原评论 ↗
引用CamperBob2 · 2026-01

“An interesting thing about transformers is that they are world-class at compressing 2D image data even when not trained on anything but textual language.”

拿它说明仅用文本训练的 Transformer 也能出色地压缩 2D 图像数据
Show HN: AI in SolidWorks原评论 ↗
引用problynought · 2026-02

“All the linguist gibberish in the typical software stack will be compressed[1] away, all the SWE middlemen unemployed.”

把语言建模视作压缩,预言软件栈里的冗余会被压缩掉
StrongDM's AI team build serious software without even looking at the code原评论 ↗
引用jackhurwitz · 2026-02

“This has also been shown to be the case with pre-trained LLMs [5].”

拿它说明预训练 LLM 的下一个 token 预测越好,无损压缩也越好
Text classification with Python 3.14's ZSTD module原评论 ↗
引用orbital-decay · 2026-06

“there's a ton of research on that, starting with the basics”

指出相关研究很多,并从这篇语言建模即压缩的基础论文说起
No, Artificial Intelligence is not Conscious - Ted Chiang原评论 ↗
引用northlondoner · 2026-09

“earlier paper, "Language Modeling Is Compression"”

引用这篇较早的论文,说明学习和语言建模就是压缩
Can gzip be a language model?原评论 ↗