“Some other things you may be interested in:”
Learning to learn by gradient descent by gradient descent
让 LSTM 学会做优化器,替代手工设计的优化算法
论文指出优化算法至今仍靠手工设计,于是把优化算法的设计转化为一个学习问题,让算法自动利用问题的结构,用 LSTM 来实现学出来的优化器。这些学出来的优化器在其训练任务上优于通用的手工设计方法,也能泛化到结构相似的新任务,实验包括简单凸问题、训练神经网络和神经风格迁移。
评论者把它当作「学习优化器 / 元学习」的经典引用,常因这个标题而被提起,用来讨论自动化算法设计,也有人询问是否有现成实现。
学习路径
读这篇之前,按顺序读书架上的 2 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出只有少数候选真正相关:先了解 Adam 这类手工设计的优化器,再看神经风格迁移这个实验任务,就能读懂论文如何让 LSTM 学成优化器并与它们对比。
- 1
论文要被取代的正是 Adam 这类手工设计的优化器,实验里也拿它做基线。
读一阶、二阶矩估计和更新规则的概念;理解手工优化器编码了哪些先验。
- 2
论文的一个实验任务是用学到的优化器做神经风格迁移,需要先懂这个任务。
读内容损失与风格损失(Gram 矩阵)的定义;只需要明白优化的是什么目标。
- 3Learning to learn by gradient descent by gradient descent
目标论文:把优化器设计转为学习问题,用 LSTM 当优化器并用梯度下降训练它。
读优化器的输入是梯度、输出是参数更新,以及按轨迹累计损失来训练;看二次函数、MNIST 和风格迁移的泛化结果。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“For now. Articles like [1] make me wonder whether this current generation of programmers might be the last.”
“Side note, anybody aware of any implementations of the "learning to learn gradient descent by gradient descent" work [0]?”
“A ML-learning group at my university recently discussed Learning to learn by gradient descent by gradient descent [1], the topic seems related.”
“The obvious citation is the wonderfully titled Learning to learn by gradient descent by gradient descent Learning to learn by gradient descent by gradient descent”
“we can now directly learn algorithms such as gradient descent[1]”
看全部 10 条 →
“if you click on Download->Other formats, you get the latex source code and in this case also the Omnigraffle files for the diagrams.”
“There has been some work related to what you are describing:”
“including "learning to optimize" approaches, e.g., Learning to learn by gradient descent by gradient descent.”
“I wonder, are you familiar with now classic paper, Learning to learn by gradient descent by gradient descent?”
- Learning to learn by gradient descent by gradient descent ↗tim_sw · 2016-06-15 · 6 分
- Learning to learn by gradient descent by gradient descent ↗sndean · 2016-06-16 · 2 分
- Learning to learn by gradient descent by gradient descent ↗tonybeltramelli · 2016-06-16 · 161 分