按年被提到
20252025 · 3 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:2501.19399 · cs.CL

Scalable-Softmax Is Superior for Attention

Ken M. Nakanishi · 2025
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2025-02-03 · 最近 2025-04-06
3不同的人
2活跃月份
3讨论串
7被引(OpenAlex)
arXiv 摘要(候选层不写导读)

The maximum element of the vector output by the Softmax function approaches zero as the input vector size increases. Transformer-based language models rely on Softmax to compute attention scores, causing the attention distribution to flatten as the context size grows. This reduces the model's ability to prioritize key information effectively and potentially limits its length generalization. To address this problem, we propose Scalable-Softmax (SSMax), which replaces Softmax in scenarios where the input vector size varies. SSMax can be seamlessly integrated into existing Transformer-based architectures. Experimental results in language modeling show that models using SSMax not only achieve faster loss reduction during pretraining but also significantly improve performance in long contexts and key information retrieval. Furthermore, an analysis of attention scores reveals that SSMax enables the model to focus attention on key information even in long contexts. Additionally, although models that use SSMax from the beginning of pretraining achieve better length generalization, those that have already started pretraining can still gain some of this ability by replacing Softmax in the attention layers with SSMax, either during or after pretraining.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论TheGradfather · 2025-02

“The OpenAI Deep Research graph showing tool calls vs pass rate reveals something fascinating about how these models handle increasing amounts of information. The relationship follows a logistic curve that plateaus around 16% pass rate, even as we allow more tool calls. This plateau behavior reflects something deeper about our current approach to AI. We'v…”

Introducing deep research原评论 ↗
评论spmurrayzzz · 2025-04

“Not sure what you mean by summary paper, its a pretty dense topic that assumes a fair amount of prior knowledge of the fundamentals. But maybe the Meta blog post may suffice for that? Otherwise yes there are lots of papers on this and related topics, a few dozen in fact. But here are some notable ones, a couple of them are linked in their blog post. RoF…”

Llama 4 Now Live on Groq原评论 ↗
被提交为 story · 1 次