按年被提到
20192023 · 3 次2026
大模型与语言模型›量化、解码与分词·arXiv:1911.02150 · cs.NE

Fast Transformer Decoding: One Write-Head is All You Need

Noam Shazeer · 2019
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2023-05-30 · 最近 2023-07-11
3不同的人
2活跃月份
2讨论串
269被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Multi-head attention layers, as used in the Transformer neural sequence model, are a powerful alternative to RNNs for moving information across and between sequences. While training these layers is generally fast and simple, due to parallelizability across the length of the sequence, incremental inference (where such paralleization is impossible) is often slow, due to the memory-bandwidth cost of repeatedly loading the large "keys" and "values" tensors. We propose a variant called multi-query attention, where the keys and values are shared across all of the different attention "heads", greatly reducing the size of these tensors and hence the memory bandwidth requirements of incremental decoding. We verify experimentally that the resulting models can indeed be much faster to decode, and incur only minor quality degradation from the baseline.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论RC_ITR · 2023-07

“>Autoregressive decoder inference is a severe bottleneck for Transformer models due to the memory bandwidth overhead from loading decoder weights and all attention keys and values at every decoding step (Shazeer, 2019; Pope et al., 2022; de Jong et al., 2022). The memory bandwidth from loading keys and values can be sharply reduced through multi-query att…”

GPT-4 Details Leaked原评论 ↗
评论why_only_15 · 2023-07

“This is the original paper: https://arxiv.org/abs/1911.02150. The idea is that with a transformer you have many heads, say 64 for LLaMa, and for each head you have 1 "query" vector one "key" vector and one "value" vector per token. Most of the cost of inferencing models is loading the key and value vectors from GPU memory to the GPU itself. the idea behi…”

GPT-4 Details Leaked原评论 ↗
被提交为 story · 1 次