按年被提到
20232023 · 3 次2026
大模型与语言模型›模型表达力与计算理论·arXiv:2311.01906 · cs.LG

Simplifying Transformer Blocks

Bobby He, Thomas Hofmann · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 4 人首次 2023-11-09 · 最近 2024-05-30
4不同的人
2活跃月份
4讨论串
25被引(OpenAlex)
arXiv 摘要(候选层不写导读)

A simple design recipe for deep Transformers is to compose identical building blocks. But standard transformer blocks are far from simple, interweaving attention and MLP sub-blocks with skip connections & normalisation layers in precise arrangements. This complexity leads to brittle architectures, where seemingly minor changes can significantly reduce training speed, or render models untrainable. In this work, we ask to what extent the standard transformer block can be simplified? Combining signal propagation theory and empirical observations, we motivate modifications that allow many block components to be removed with no loss of training speed, including skip connections, projection or value parameters, sequential sub-blocks and normalisation layers. In experiments on both autoregressive decoder-only and BERT encoder-only models, our simplified transformers emulate the per-update training speed and performance of standard transformers, while enjoying 15% faster training throughput, and using 15% fewer parameters.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论skinner_ · 2023-11

“There's this nice new paper ("Simplifying Transformer Blocks" https://arxiv.org/abs/2311.01906) about getting rid of all the unnecessary parts of a transformer. On their dataset, they manage to get rid of the layernorm, the skip connection, the value matrix, and the projection matrix.”

Q-Transformer原评论 ↗
评论lukemerrick · 2024-05

“Just skimmed so far and didn't see any reference to the Simplified Transformer block of https://arxiv.org/abs/2311.01906 (and it seems they also left out grouped query attention, too, as pointed out by another comment). While lazy me wants them to explain how their approach compares to these approaches, it looks like their exposition is pretty clear (qui…”

New attention mechanisms that outperform standard multi-head attention原评论 ↗
被提交为 story · 2 次