按年被提到
20242024 · 7 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:2404.07143 · cs.CL

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

Tsendsuren Munkhdalai, Manaal Faruqui, Siddharth Gopal · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 7 人首次 2024-04-11 · 最近 2024-05-30
7不同的人
2活跃月份
7讨论串
78被引(OpenAlex)
arXiv 摘要(候选层不写导读)

This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. Our approach introduces minimal bounded memory parameters and enables fast streaming inference for LLMs.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论HarHarVeryFunny · 2024-04

“Scaling alone will certainly not get us to AGI, but that's trivially obvious. The "scale is all you need" argument, from anyone intelligent, assumes that other obvious deficiencies such as lack of short-term memory (to enable more capable planning/reasoning) will also be taken care of along the way, and I'd not be surprised to see that particular one add…”

Scaling will never get us to AGI原评论 ↗
评论ttul · 2024-04

“https://arxiv.org/html/2404.07143v1 “This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation””

Ask HN: Is RAG the Future of LLMs?原评论 ↗
评论underlines · 2024-04

“tons and tons of papers, most of them had some disadvantages. Can't have the cake and eat it too: https://arxiv.org/html/2404.08801v1 Meta Megalodon https://arxiv.org/html/2404.07143v1 Google Infini-Attention https://arxiv.org/html/2402.13753v1 LongRoPE and a ton more”

Mixtral 8x22B原评论 ↗
评论westurner · 2024-05

“Because self-attention can be replaced with FFT for a loss in accuracy and a reduction in kWh [1], I suspect that the Quantum Fourier Transform can also be substituted for attention in LLMs. [1] https://syncedreview.com/2021/05/14/deepmind-podracer-tpu-based-rl-frameworks-deliver-exceptional-performance-at-low-cost-19/ "You Need to Pay Better Attention"…”

New attention mechanisms that outperform standard multi-head attention原评论 ↗
被提交为 story · 3 次