按年被提到
20262026 · 3 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:2601.05280 · cs.IT

Large Language Models are Shannon Lossy Compressors Not Solomonoff Induction Estimators: Self-improvement and Singularity Are Not Near Without Symbolic Model Synthesis

Hector Zenil, Abicumaran Uthamacumaran, Luan Ozelim · 2026
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2026-04-27 · 最近 2026-06-05
3不同的人
2活跃月份
3讨论串
2被引(OpenAlex)
arXiv 摘要(候选层不写导读)

We connect two questions in Algorithmic Information Theory (AIT), Machine Learning (ML) and Artificial General Intelligence (AGI): whether LLMs estimate Solomonoff induction, and whether they can self-improve towards an AI Singularity. We provide theoretical, methodological and empirical answers in the negative but show how limits can be circumvented. Cross-entropy, negative log-likelihood and related next-token objectives cannot alone implement Solomonoff induction: they fit supplied conditionals rather than a program-weighted universal mixture. More computation can improve fit within a fixed objective but cannot change its inductive principle without external hyperparameter or architectural tuning; they alone do not deliver Solomonoff-Levin optimal prediction. For finite learners and observers, theoretical boundaries become less decisive and approaches diverge. Resource-bounded estimators are finite mechanism-search tools whose divergence does not violate algorithmic information conservation. All 26 served language-model checkpoints across five pre-training families, 0.8-35 billion parameters and 1.9-8.5 bits per weight, evaluated at their commitments over a closed alphabet, violate the dominance guarantee defining a universal mixture. Against a 3.32-bit bound attained by a genuine mixture, the best model trails a Krichevsky-Trofimov code by 4.5 bits, the median by 36 and the worst by 128; excess grows to every stream's end rather than settling to a constant. Served conditionals fail to form a mixture over the declared class in 79 of 91 checkpoint-designs; neither scale nor post-training closes the gap. Frontier developers adopt neurosymbolic approaches, including Fable and Astra, incorporating model synthesis via neurosymbolic computation. They are no longer purely statistical LLMs, making them better, though still limited, candidates for higher forms of induction & model synthesis.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论anthk · 2026-04

“Good. Let the slopwares collapse into themselves, from GNU/Linux, to Hurd (sadly) and Ubuntu. Trisquel will be damned, but Hyperbola BSD -after Hyperbola GNU- will be like the Phoenix bird. https://arxiv.org/html/2601.05280v2”

Ubuntu Linux Will Begin Landing AI Features Throughout the Next Year原评论 ↗
评论mlmonkey · 2026-06

“I don't know how RSI aligns with DPI ("Data Processing Inequality", which states that, basically, unless you have an infinite supply of real data, you will suffer from model collapse). Models can't keep improving themselves infinitely. See, for example, https://arxiv.org/html/2601.05280v2”

Sakana AI's Recursive Self-Improvement (RSI) Lab原评论 ↗
被提交为 story · 1 次