“For a more rigorous explanation, see Identifying and attacking the saddle point problem in high-dimensional non-convex optimization”
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
高维非凸优化的难点在鞍点而非局部极小,并提出 saddle-free Newton 法
作者结合统计物理、随机矩阵理论和神经网络理论的结果以及实验证据,论证在高维问题中真正的困难来自大量鞍点,而不是误差很高的局部极小值;鞍点周围的高误差平台会显著拖慢学习,还会制造局部极小的假象。基于此提出二阶优化方法 saddle-free Newton,能比梯度下降和拟牛顿法更快逃离鞍点,并在深层和循环神经网络训练上给出数值证据。
当有人担心神经网络会卡在局部极小值时,评论者常引用它说明在高维空间里更大的障碍是鞍点而不是局部极小。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Check any of the literature on this subject:”
“this 2014 paper was pretty big for tackling it”
“For more details, check out Identifying and attacking the saddle point problem in high-dimensional non-convex optimization”
“In fact, in high dimensions, saddle points are a much bigger problem than local minima. [0]”
“cf Identifying and attacking the saddle point problem in high-dimensional non-convex optimization”
看全部 8 条 →
“This paper makes the points that it's the saddles and not local minima that are the problem:”
“A related viewpoint is that overparametrization is good because the model is stranded when the Hessian has all positive/zero eigenvalues.”