推荐thesz · 2025-04
“I really enjoyed that paper, a gem that puts light everywhere.”
推荐这篇论文,称它是一颗照亮各处的宝石,用多项式拟合重现 double descent 与记忆
用软归纳偏置和 PAC-Bayes 解释深度学习的泛化之谜
论文认为良性过拟合、双下降和过参数化成功等看似反常的泛化现象并非神经网络独有,也没有那么神秘,可以用 PAC-Bayes 和可数假设空间界等经典泛化框架直观理解并严格刻画。核心思想是软归纳偏置:不去限制假设空间,而是保持灵活的假设空间,同时对与数据一致的更简单解保持软偏好。作者同时指出,深度学习在表示学习、模式连通性和相对通用性等方面确有特别之处。
HN 上的人在争论深度学习泛化是否神秘时推荐它,用来说明多项式拟合也能复现双下降,以及现有学习理论哪些能解释、哪些不能。
“I really enjoyed that paper, a gem that puts light everywhere.”
“Check out Andrew Gordon Wilson's excellent paper”
“There are strong arguments that deep learning generalization (and robustness in a slightly different sense) can be successfully explained in terms of PAC-Bayes theory, for example: Deep Learning is Not So Mysterious or Different”
“Reminded me strongly of the paper”
“Also see the argument around figure 3 here.”