按年被提到
20252026 · 2 次2026
机器学习与深度学习›深度学习理论与泛化·arXiv:2507.17912 · cs.LG

SETOL: A Semi-Empirical Theory of (Deep) Learning

Charles H Martin, Christopher Hinrichs · 2025
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2025-07-25 · 最近 2026-02-04
2不同的人
2活跃月份
3讨论串
—被引(OpenAlex)
arXiv 摘要(候选层不写导读)

We present a SemiEmpirical Theory of Learning (SETOL) that explains the remarkable performance of State-Of-The-Art (SOTA) Neural Networks (NNs). We provide a formal explanation of the origin of the fundamental quantities in the phenomenological theory of Heavy-Tailed Self-Regularization (HTSR): the heavy-tailed power-law layer quality metrics, alpha and alpha-hat. In prior work, these metrics have been shown to predict trends in the test accuracies of pretrained SOTA NN models, importantly, without needing access to either testing or training data. Our SETOL uses techniques from statistical mechanics as well as advanced methods from random matrix theory and quantum chemistry. The derivation suggests new mathematical preconditions for ideal learning, including a new metric, ERG, which is equivalent to applying a single step of the Wilson Exact Renormalization Group. We test the assumptions and predictions of SETOL on a simple 3-layer multilayer perceptron (MLP), demonstrating excellent agreement with the key theoretical assumptions. For SOTA NN models, we show how to estimate the individual layer qualities of a trained NN by simply computing the empirical spectral density (ESD) of the layer weight matrices and plugging this ESD into our SETOL formulas. Notably, we examine the performance of the HTSR alpha and the SETOL ERG layer quality metrics, and find that they align remarkably well, both on our MLP and on SOTA NNs.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论riemannzeta · 2026-02

“Not an expert in neurobiology, but... heck this is Hacker News: Let's assume that the brain implements a renormalization group flow in a way similar to transformers, as described here: https://arxiv.org/abs/2507.17912 In biological neural networks, synaptic weight distributions presumably have some effective α. Learning perturbs α away from optimal — ov…”

Once Thought to Support Neurons, Astrocytes Turn Out to Be in Charge原评论 ↗
评论riemannzeta · 2026-02

“Neat result. The symmetry exploitation here reminds me of recent work connecting neural network training dynamics to renormalization group theory. Charles Martin's SETOL paper https://arxiv.org/abs/2507.17912 shows that well-trained layers converge to something like an RG fixed point—the eigenvalue spectrum of the weight matrix develops power-law tails wi…”

Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation原评论 ↗
被提交为 story · 1 次