“It is missing a proper evaluation section. Specifically, I wonder how this compares with https://arxiv.org/abs/2401.03462 (Activation Beacon)”
Long Context Compression with Activation Beacon
Long context compression is a critical research problem due to its significance in reducing the high computational and memory costs associated with LLMs. In this paper, we propose Activation Beacon, a plug-in module for transformer-based LLMs that targets effective, efficient, and flexible compression of long contexts. To achieve this, our method introduces the following technical designs. 1) We directly compress the activations (i.e. keys and values at every layer), rather than leveraging soft prompts to relay information (which constitute a major bottleneck to encapsulate the complex information within long contexts). 2) We tailor the compression workflow, where each fine-grained input unit is progressively compressed, enabling high-quality compression and efficient computation during both training and inference. 3) We train the model through compression-based auto-regression, making full use of plain texts and instructional data to optimize the model's compression performance. 4) During training, we randomly sample a compression ratio at each step, teaching the model to support a wide range of compression configurations. Extensive evaluations are conducted on various long-context tasks whose lengths (e.g., 128K) may far exceed the maximum training length (20K), such as document understanding, few-shot learning, and Needle-in-a-Haystack. Whilst existing methods struggle to handle these challenging tasks, Activation Beacon maintains a comparable performance to the uncompressed baseline across various scenarios, achieving a 2x acceleration in inference time and an 8x reduction of memory costs for KV cache. Our data, model, and code have been released at \url{https://github.com/FlagOpen/FlagEmbedding/}.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及- Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon ↗bx376 · 2024-01-11 · 2 分
- Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon ↗belter · 2024-01-15 · 1 分
- Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon ↗PaulHoule · 2024-01-16 · 1 分
- Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon ↗endymi0n · 2024-01-30 · 4 分