按年被提到
20262026 · 7 次2026
机器学习与深度学习›贝叶斯方法与 AutoML·arXiv:2601.03220 · cs.LG

From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence

Marc Finzi, Shikai Qiu, Yiding Jiang, Pavel Izmailov, J. Zico Kolter, Andrew Gordon Wilson · 2026
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 7 人首次 2026-01-08 · 最近 2026-03-23
7不同的人
2活跃月份
6讨论串
8被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Can we learn more from data than existed in the generating process itself? Can new and useful information be constructed from merely applying deterministic transformations to existing data? Can the learnable content in data be evaluated without considering a downstream task? On these questions, Shannon information and Kolmogorov complexity come up nearly empty-handed, in part because they assume observers with unlimited computational capacity and do not target the useful information content. In this work, we identify and exemplify three seeming paradoxes in information theory: (1) information cannot be increased by deterministic transformations; (2) information is independent of the order of data; (3) likelihood modeling is merely distribution matching. To shed light on the tension between these results and modern practice, and to quantify the value of data, we introduce epiplexity, a formalization of information capturing what computationally bounded observers can learn from data. Epiplexity captures the structural content in data while excluding time-bounded entropy, the random unpredictable content exemplified by pseudorandom number generators and chaotic dynamical systems. With these concepts, we demonstrate how information can be created with computation, how it depends on the ordering of the data, and how likelihood modeling can produce more complex programs than present in the data generating process itself. We also present practical procedures to estimate epiplexity which we show capture differences across data sources, track with downstream performance, and highlight dataset interventions that improve out-of-distribution generalization. In contrast to principles of model selection, epiplexity provides a theoretical foundation for data selection, guiding how to select, generate, or transform data for learning systems.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论kingstnap · 2026-01

“Related: https://arxiv.org/abs/2601.03220 This is a paper that recently got popular ish and discusses the counter to your viewpoint. > Paradox 1: Information cannot be increased by deterministic processes. For both Shannon entropy and Kolmogorov complexity, deterministic transformations cannot meaningfully increase the information content of an object. …”

Project Genie: Experimenting with infinite, interactive worlds原评论 ↗
评论est · 2026-03

“> the architect → developer → reviewer pipeline actually produces better results than just... talking to one strong model in one session? There's a 63 page paper with mathematical proof if you really into this. https://arxiv.org/html/2601.03220v1 My takeaway: AI learns from real-world texts, and real-world corpus are used to have a role split of archit…”

"How I write software with LLMs"原评论 ↗
评论codeflo · 2026-03

“>> the architect → developer → reviewer pipeline actually produces better results than just... talking to one strong model in one session? > There's a 63 page paper with mathematical proof if you really into this. > https://arxiv.org/html/2601.03220v1 I'm confused. The linked paper is not primarily a mathematics paper, and to the extent that it is, pro…”

"How I write software with LLMs"原评论 ↗
评论Noumenon72 · 2026-03

“This presumes that the value was created by the authors and not the people who found a way to use the structure in the training set to create intelligence. Like crediting wi-fi router owners with creating Wi-Fi Positioning System instead of the people who realized they could wardrive around to create maps and extract a new kind of value that would never h…”

You Are Not Your Job原评论 ↗
被提交为 story · 3 次