“Maybe Flash attention has a shot.”
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
用分块减少 GPU 显存读写的 IO 感知精确注意力算法
自注意力的时间和显存复杂度随序列长度呈二次增长,近似注意力方法往往牺牲质量却没有换来实际加速。论文认为缺失的原则是让注意力算法具备 IO 感知,即考虑 GPU 各级内存之间的读写,并提出 FlashAttention:用分块(tiling)减少 GPU 高带宽显存(HBM)与片上 SRAM 之间的读写次数,经分析需要的 HBM 访问比标准注意力更少,并在一定范围的 SRAM 大小下最优。它还可扩展成块稀疏注意力。训练上,BERT-large 比 MLPerf 1.1 记录端到端快 15%,GPT-2 快 3 倍,长程竞技场快 2.4 倍,并带来更长上下文和更高质量。
评论者把它当作缓解注意力二次方成本的可行方案和现代 LLM 训练的常用组件来引用,也有人指出其主要局限是需要针对每种 GPU 架构手写 CUDA 内核。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Maybe Flash attention has a shot.”
“FlashAttention - Section 5 suggests that the biggest limitation is that custom CUDA kernels need to be coded on a per-GPU architecture basis.”
“The models were trained with Flash Attention, Xpos Rotary Embeddings for better length extrapolation, and multi-query single-key-value attention for more efficient decoding: FlashAttention”
“I would still have expected this paper to at least mention FlashAttention [1] since they both leverage flash memory.”
“FlashAttention(2)[0] reduces context-length space complexity to linear.”
看全部 8 条 →
“Specifically: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.”
“OpenAI FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness: Tri Dao et al., 2022, FlashAttention”
- Flash Attention accelerates GPT2 3x ↗sinenomine · 2022-06-01 · 2 分
- Flash Attention ↗fzliu · 2023-04-20 · 1 分
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness ↗ftxbro · 2023-05-05 · 1 分
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness ↗TaurenHunter · 2024-01-16 · 1 分