“This is one paper you could read”
Language Modeling Is Compression
从压缩视角看语言模型,Chinchilla 压缩图像胜过 PNG
论文指出预测模型与无损压缩器可以互相转换,主张用压缩的视角看待预测问题,并评估大型基础模型的压缩能力。例如主要用文本训练的 Chinchilla 70B 能把 ImageNet 图像块压到原大小的 43.4%、LibriSpeech 音频压到 16.4%,优于 PNG(58.5%)和 FLAC(30.3%);压缩视角还对缩放定律、分词和上下文学习提供新的见解,任何压缩器(如 gzip)也可用来构造条件生成模型。
评论者把它当作「语言建模即压缩」的出处来引用,用来论证 LLM 本质上是无损压缩,以及仅用文本训练的 Transformer 也能压缩图像。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“It goes along the same lines as the recent paper from Deepmind stating that DL (language modeling in their case) is compression.”
“I recall there was a paper with a relevant title recently… Language Modeling Is Compression”
“Many papers on this, recent one”
“The increasing recognition that "Language Modeling Is Compression" Language Modeling Is Compression has not yet been accompanied by recognition”
“All we’re doing is engineering new data compression and retrieval techniques: Language Modeling Is Compression”
看全部 20 条 →
“He just needed to use LLMs instead: Language Modeling Is Compression”
“Oh well, you should tell the folks at DeepMind and Meta about these objective facts”
“as we continue to compress from the model all the non-essential syntax sugar of modern software.”
“Edit: Language Modeling Is Compression”
“I look forward to continuing my work in the hardware world to further compress and reduce the useless state of past systems of though we copy paste around to serve developers, to reduce context to sort through, and improve model quality: Language Modeling Is Compression”
“are essentially compression algorithms?”
“Not to mention, Language Modeling is Compression Language Modeling Is Compression”
“Language modeling is compression, whittle down graph to reduce duplication and data with little relationship: Language Modeling Is Compression”
“LLMs are a form of compression [1].”
“An interesting thing about transformers is that they are world-class at compressing 2D image data even when not trained on anything but textual language.”
“All the linguist gibberish in the typical software stack will be compressed[1] away, all the SWE middlemen unemployed.”
“This has also been shown to be the case with pre-trained LLMs [5].”
“there's a ton of research on that, starting with the basics”
“earlier paper, "Language Modeling Is Compression"”