“I think other than the title being a bit misleading, the paper is good. I say misleading because they replace Layer Normalization with a tanh function, which still bounds the range to [-1,1]. Plenty of people would call that normalization (an unfortunately overloaded term). While the result isn't too surprising it has a good ablation study and helps buil…”
Transformers without Normalization
Normalization layers are ubiquitous in modern neural networks and have long been considered essential. This work demonstrates that Transformers without normalization can achieve the same or better performance using a remarkably simple technique. We introduce Dynamic Tanh (DyT), an element-wise operation $DyT($x$) = \tanh(α$x$)$, as a drop-in replacement for normalization layers in Transformers. DyT is inspired by the observation that layer normalization in Transformers often produces tanh-like, $S$-shaped input-output mappings. By incorporating DyT, Transformers without normalization can match or exceed the performance of their normalized counterparts, mostly without hyperparameter tuning. We validate the effectiveness of Transformers with DyT across diverse settings, ranging from recognition to generation, supervised to self-supervised learning, and computer vision to language models. These findings challenge the conventional understanding that normalization layers are indispensable in modern neural networks, and offer new insights into their role in deep networks.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
没有评论直接链到它;下面是它被提交的讨论串里回复最多的评论(不计入统计)- Transformers Without Normalization ↗qianli_cs · 2025-03-14 · 4 分
- Transformers Without Normalization ↗Anon84 · 2025-03-15 · 2 分
- Transformers Without Normalization ↗fzliu · 2025-03-16 · 2 分
- Transformers without normalization ↗kaycebasques · 2025-07-24 · 42 分