按年被提到
20252025 · 2 次2026
机器学习与深度学习›贝叶斯方法与 AutoML·arXiv:2507.11768 · stat.ML

LLMs are Bayesian, In Expectation, Not in Realization

Leon Chlon, Fatima Sheaib, Zein Khamis, Maggie Chlon, Mahdi El Zein, MarcAntonio M. Awada · 2025
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2025-07-24 · 最近 2025-09-09
2不同的人
2活跃月份
2讨论串
4被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Bayesian accounts of in-context learning face a direct objection: exact posterior predictives for exchangeable data are invariant to task-preserving order, yet transformers change next-token probabilities when the same examples are serialized differently. We show this objection targets a structural invariant rather than the quantity scoring online prediction. For any Bayesian reference, excess prequential code length is exactly cumulative predictive KL. For unordered support sets that must be serialized, the expected regret of a single admissible ordering decomposes into that of the order-averaged predictor plus an order-averaging gain. Exchangeability violations are therefore not binary refutations; they are priced by log loss. We instantiate the theory with KT/Dirichlet finite-alphabet prediction and coarsened Bayesian linear-regression (BLR) predictive distributions. On Qwen2.5-7B/14B, floored candidate distributions at support $256$ have one-step excess code lengths of $0.020/0.011$ bits for Bernoulli and $0.039/0.022$ bits for four-way categorical prediction, with candidate mass above $0.999$; coarsened BLR continuations increasingly match the posterior-predictive digit distribution as support grows. A frequentist plug-in baseline sharpens the reading: the predictive distributions sit closer to the Bayesian posterior predictive than to the maximum-likelihood plug-in, by a margin largest at small support, where the plug-in is degenerate, and vanishing as the references converge. Position interventions and a from-scratch ablation localize order sensitivity to the positional encoding, activation patching tests causal use of decoded sufficient statistics, and permutation mixtures quantify the downstream log-loss cost of arbitrary orderings. Transformers need not realize exchangeable posterior predictives for every serialization to be Bayes-competitive prequential predictors.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论spindump8930 · 2025-09

“This topic is interesting, but the repo and paper have a lot of inconsistencies that make me think this work is hiding behind lots of dense notation and language. For one, the repo states: > This implementation follows the framework from the paper “Compression Failure in LLMs: Bayesian in Expectation, Not in Realization” (NeurIPS 2024 preprint) and relat…”

Hallucination Risk Calculator原评论 ↗
被提交为 story · 1 次