按年被提到
20232023 · 2 次2026
大模型与语言模型›模型表达力与计算理论·arXiv:2310.01889 · cs.CL

Ring Attention with Blockwise Transformers for Near-Infinite Context

Hao Liu, Matei Zaharia, Pieter Abbeel · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2023-10-04 · 最近 2024-02-16
3不同的人
2活跃月份
3讨论串
100被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Transformers have emerged as the architecture of choice for many state-of-the-art AI models, showcasing exceptional performance across a wide range of AI applications. However, the memory demands imposed by Transformers limit their ability to handle long sequences, thereby posing challenges in utilizing videos, actions, and other long-form sequences and modalities in complex environments. We present a novel approach, Ring Attention with Blockwise Transformers (Ring Attention), which leverages blockwise computation of self-attention and feedforward to distribute long sequences across multiple devices while fully overlapping the communication of key-value blocks with the computation of blockwise attention. Our approach enables training and inference of sequences that are up to device count times longer than those achievable by prior memory-efficient Transformers, without resorting to approximations or incurring additional communication and computation overheads. Extensive experiments on language modeling and reinforcement learning tasks demonstrate the effectiveness of our approach in allowing millions of tokens context size and improving performance.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论earth2mars · 2023-10

“It would be interesting to see how does ring attention technique affect this. This maybe still valid for cost reasons, but unlimited context is like in-memory computing vs. traditional. https://arxiv.org/pdf/2310.01889.pdf”

MemGPT: Towards LLMs as Operating Systems原评论 ↗
评论xiphias2 · 2024-02

“They are using RingAttention that scales self attention computation linearly by number of devices by passing KV result blocks in a ring: https://arxiv.org/abs/2310.01889 (Submitted on 3 Oct 2023)”

LWM – Open LLM with 1M Tokens Context Window原评论 ↗
被提交为 story · 1 次