“The full title of the paper is “Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis.””
Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis
对比演化网络与 SGD 网络,提出 SGD 训练得到“破碎纠缠表征”。
这篇立场论文质疑“性能更好必然意味着内部表征更好”的表征乐观主义。作者在生成单张图像这一极简任务上,对比了通过开放式搜索演化出的网络与常规 SGD 训练的网络:两者输出行为相同,内部表征却差别很大。SGD 网络出现一种称为“破碎纠缠表征”(FER)的混乱,而演化网络基本没有 FER,甚至接近统一的因子化表征(UFR)。作者认为在大模型中 FER 可能损害泛化、创造力和(持续)学习,理解并缓解它对表征学习的未来可能至关重要。
评论者借它论证梯度下降得不到功能上有组织的内部表征,把“破碎纠缠表征”当作对深度学习表征质量的批评来引用,也有人称其为 Ken Stanley 最新的表征论文。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Stochastic gradient descent does not yield functional internal representations[1]; it fills the bucket of parameters with one distribution and one distribution only.”
“Survival is not a goal, it is a constraint.”
“This reminds me of the latest Ken Stanley paper on representations.”
“maybe the more modern lens on this is e.g. Questioning Representational Optimism in Deep Learning.”
“which is hopefully coherent and consistent -- but definitely not always! see, e.g., Questioning Representational Optimism in Deep Learning)”
- Questioning Representational Optimism in Deep Learning ↗artninja1988 · 2025-05-20 · 1 分
- Questioning Representational Optimism in Deep Learning ↗publicdaniel · 2025-06-05 · 1 分
- Questioning Representational Optimism in Deep Learning ↗vatsachak · 2025-11-15 · 1 分