“And btw that's a good paper. Probably one of the most satisfying DL papers I've read in recent years.”
Neural Networks and the Chomsky Hierarchy
用乔姆斯基层级预测不同神经网络架构在分布外泛化上的极限
作者做了大规模实证研究(20,910 个模型、15 个任务),检验计算理论能否预测神经网络泛化的实际极限,发现按乔姆斯基层级对任务分组可以预测某些架构能否泛化到分布外输入,包括数据和训练时间再多也无法出现任何非平凡泛化的负面结果。结果显示在其任务子集上,RNN 和 Transformer 无法在非正则任务上泛化,LSTM 能解决正则和计数器语言任务,只有带结构化记忆(如栈或记忆带)的网络才能成功泛化到上下文无关和上下文相关任务。
评论者讨论 Transformer 的局限时反复引用它,作为 Transformer 在嵌套结构等乔姆斯基层级任务上不如 LSTM 或带记忆网络的形式化证据,也有人提醒结论并不绝对,因为训练不是无限进行的。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“worth exploring the work discussed in the paper”
“This is a very interesting article, in particular because it brings up that whole Chomsky topic of the grammar hierarchy (see Neural Networks and the Chomsky Hierarchy).”
“Some big limitations of transformers as well:”
“Recurrency will help, see the empirical results in this paper: Neural Networks and the Chomsky Hierarchy.”
“There are also limits to the sequence to sequence things they can effectively learn, in some cases weaker than LSTMs:”
看全部 13 条 →
“This limitation has been pointed out formally by the DeepMind paper "Neural Networks and the Chomsky Hierarchy".”
“This limitation has been pointed out formally by the DeepMind paper "Neural Networks and the Chomsky Hierarchy".”
“Deep Mind had this paper on that, showing weaknesses of transformers relative to other models in generalizing from example productions from different tiers of the hierarchy:”
“It's not conclusive since it is not training indefinitely”
“transformers get beat out on learning to reproduce outs towards the general computer end”
“Transformers, even equipped with memory, can only represent Regular automata [1].”
“This deepmind paper showed big limitations on nested structures with transformers, LSTMs actually did better:”
- Neural Networks and the Chomsky Hierarchy by DeepMind ↗ashvardanian · 2022-07-16 · 5 分
- Neural Networks and the Chomsky Hierarchy [DeepMind, 2022] ↗cma · 2023-01-23 · 2 分