按年被提到
20242025 · 2 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:2412.06464 · cs.CL

Gated Delta Networks: Improving Mamba2 with Delta Rule

Songlin Yang, Jan Kautz, Ali Hatamizadeh · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2025-02-15 · 最近 2025-06-17
2不同的人
2活跃月份
2讨论串
28被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Linear Transformers have gained attention as efficient alternatives to standard Transformers, but their performance in retrieval and long-context tasks has been limited. To address these limitations, recent work has explored two distinct mechanisms: gating for adaptive memory control and the delta update rule for precise memory modifications. We observe that these mechanisms are complementary: gating enables rapid memory erasure while the delta rule facilitates targeted updates. Building on this insight, we introduce the gated delta rule and develop a parallel training algorithm optimized for modern hardware. Our proposed architecture, Gated DeltaNet, consistently surpasses existing models like Mamba2 and DeltaNet across multiple benchmarks, including language modeling, common-sense reasoning, in-context retrieval, length extrapolation, and long-context understanding. We further enhance performance by developing hybrid architectures that combine Gated DeltaNet layers with sliding window attention or Mamba2 layers, achieving both improved training efficiency and superior task performance.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论korbip · 2025-02

“There is a LOT of effort in the research community currently: 1. Improving the Self-Attention in the Transformer as is, keeping the quadratic complexity, which has some theoretical advantage in principle[1]: The most hyped one probably DeepSeek's Multi-Latent Attention[15], which kind of is Attention still - but also somehow different. 2. Linear RNNs: T…”

Ask HN: Is anybody building an alternative transformer?原评论 ↗
评论juank10 · 2025-06

“Funnily enough, the code was deleted in the repo, but can still be seen in the commits. It's what you would expect from the paper:D On the general topic of non-attention LLMs, I recommend checking out the MesaNet [1], Rodimus [2], Gated DeltaNet [3], or Mamba2 [4]. They are currently SOTA. However, I have yet to see a compelling non attention based mod…”

Breaking Quadratic Barriers: A Non-Attention LLM for Ultra-Long Context Horizons原评论 ↗