按年被提到
20242026 · 2 次2026
大模型与语言模型›模型表达力与计算理论·arXiv:2410.18613 · cs.LG

Rethinking Attention: Polynomial Alternatives to Softmax in Transformers

Hemanth Saratchandran, Jianqiao Zheng, Yiping Ji, Wenbo Zhang, Simon Lucey · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-10-27 · 最近 2026-10-17
2不同的人
3活跃月份
3讨论串
4被引(OpenAlex)
arXiv 摘要(候选层不写导读)

This paper questions whether the strong performance of softmax attention in transformers stems from producing a probability distribution over inputs. Instead, we argue that softmax's effectiveness lies in its implicit regularization of the Frobenius norm of the attention matrix, which stabilizes training. Motivated by this, we explore alternative activations, specifically polynomials, that achieve a similar regularization effect. Our theoretical analysis shows that certain polynomials can serve as effective substitutes for softmax, achieving strong performance across transformer applications despite violating softmax's typical properties of positivity, normalization, and sparsity. Extensive experiments support these findings, offering a new perspective on attention mechanisms.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论D-Machine · 2026-01

“1. I do not think it is orthogonal, but, regardless, there is plenty of research trying to get explainability out of all aspects of scaled dot-product attention layers (weights, QKV projections, activations, other aspects), and trying to explain deep models generally via sort of bottom-up mechanistic approaches. I think it can be clearly argued this does …”

The Q, K, V Matrices原评论 ↗
评论D-Machine · 2026-10

“Yes, the main problem with this whole way of thinking about transformers and self-attention is that when you look at how they are actually implemented (i.e. what Q, K, and V are applied to at each layer, and consider the loss function during training), then the analogy to memory quickly starts to look tenuous and far-removed from what is actually happenin…”

You could have invented transformers from memory原评论 ↗
被提交为 story · 1 次