按年被提到
20242024 · 1 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:2409.15355 · cs.LG

Block-Attention for Efficient Prefilling

Dongyang Ma, Yan Wang, Lan Tian · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-10-06 · 最近 2025-10-12
2不同的人
2活跃月份
2讨论串
6被引(OpenAlex)
arXiv 摘要(候选层不写导读)

We introduce Block-attention, an attention mechanism designed to address the increased inference latency and cost in Retrieval-Augmented Generation (RAG) scenarios. Traditional approaches often encode the entire context in an auto-regressive manner. Instead, Block-attention divides retrieved documents into discrete blocks, with each block independently calculating key-value (KV) states except for the final block. In RAG scenarios, by defining each passage as a block, Block-attention enables us to reuse the KV states of passages that have been seen before, thereby significantly reducing the latency and the computation overhead during inference. The implementation of Block-attention involves block segmentation, position re-encoding, and fine-tuning the LLM to adapt to the Block-attention mechanism. Experiments on 11 diverse benchmarks, including RAG, ICL, and general domains, demonstrate that after block fine-tuning, the Block-attention model not only achieves performance comparable to that of full-attention models, but can also seamlessly switch between the block and full attention modes without any performance loss. Notably, Block-attention significantly reduces the time to first token (TTFT) and floating point operations (FLOPs) to a very low level. It only takes 45 ms to output the first token for an input sequence with a total length of 32K. Compared to the full-attention models, the TTFT and corresponding FLOPs are reduced by 98.7% and 99.8%, respectively. Additionally, in Appendix A, we elaborate on how Block-attention is applied in Game AI scenario and the substantial potential benefits it entails. We strongly suggest researchers in the gaming field not to overlook this section.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论Palmik · 2025-10

“The observation about the "block-diagonal patterns" in RAG isn't new and has been exploited / explored before: - https://arxiv.org/abs/2410.07590 (literally titled "Block-Attention for Efficient RAG") - https://arxiv.org/abs/2409.15355v3 - https://arxiv.org/abs/2212.10947 The REFRAG paper does not cite any of these.”

Meta Superintelligence Labs' first paper is about RAG原评论 ↗
被提交为 story · 1 次