“optimizers like ADAM have explicit guarantees about things like convergence.”
Adam: A Method for Stochastic Optimization
基于自适应矩估计的一阶随机优化算法 Adam
论文提出 Adam,一种基于低阶矩自适应估计的一阶随机梯度优化算法,实现简单、计算高效、内存占用小,对梯度的对角缩放不变,适合大规模数据和参数,也适用于非平稳目标以及含噪声、稀疏的梯度。作者在在线凸优化框架下给出了遗憾界,实验显示其效果优于其他随机优化方法,并讨论了基于无穷范数的变体 AdaMax。
评论者多把它当作史上被引最多的 AI 论文之一和默认优化器的出处,并提到它无需经过期刊发表就获得了巨大关注,也有人说它可读性好但收敛性证明较复杂。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“appears to be a well-written introduction to stuff variational autoencoders [2] and ADAM [3].”
“Adam the optimization algo you might need (and even there I am not sure). And it is very readable.”
“are the authors of the Adam Paper, Adam”
“Kingma is most notable for writing one of the most cited papers in AI.”
“Something not being published in a journal/conference proceedings clearly does not prevent it from drawing tons of research attention, as we saw in numerous cases like the Adam optimizer [1].”
看全部 7 条 →
“developed Adam [0] well before it was flashy and profitable.”
- Adam's method for stochastic optimization ↗Nimsical · 2015-10-13 · 1 分