按年被提到
20252025 · 3 次2026
大模型与语言模型›混合专家与高效训练·arXiv:2503.01868 · cs.LG

Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale

Jerome Ku, Eric Nguyen, David W. Romero, Garyk Brixi, Brandon Yang, Anton Vorontsov, Ali Taghibakhshi, Amy X. Lu, Dave P. Burke, Greg Brockman, Stefano Massaroli, Christopher Ré 等 16 人 · 2025
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2025-03-05 · 最近 2025-04-29
2不同的人
2活跃月份
3讨论串
9被引(OpenAlex)
arXiv 摘要(候选层不写导读)

We introduce convolutional multi-hybrid architectures, with a design grounded on two simple observations. First, operators in hybrid models can be tailored to token manipulation tasks such as in-context recall, multi-token recall, and compression, with input-dependent convolutions and attention offering complementary performance. Second, co-designing convolution operators and hardware-aware algorithms enables efficiency gains in regimes where previous alternative architectures struggle to surpass Transformers. At the 40 billion parameter scale, we train end-to-end 1.2 to 2.9 times faster than optimized Transformers, and 1.1 to 1.4 times faster than previous generation hybrids. On H100 GPUs and model width 4096, individual operators in the proposed multi-hybrid StripedHyena 2 architecture achieve two-fold throughput improvement over linear attention and state-space models. Multi-hybrids excel at sequence modeling over byte-tokenized data, as demonstrated by the Evo 2 line of models. We discuss the foundations that enable these results, including architecture design, overlap-add blocked kernels for tensor cores, and dedicated all-to-all and point-to-point context parallelism strategies.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论jwilber · 2025-04

“Achieved by “applying convolution operations over queries, keys and heads, allowing nearby queries and keys to affect each other's attention weights for more precise attention” Cool to see convolutions making such a comeback lately in the llm world. See also the recent striped hyena2 architecture, which uses the conv-based hyena operator to great success…”

Multi-Token Attention原评论 ↗
评论jwilber · 2025-04

“LLM/state space models have been popular for some years now, see: https://arxiv.org/abs/2212.14052 More recently, hybrid architectures that utilize attention plus other operators are gaining traction. See https://arxiv.org/abs/2503.01868”

Bamba: An open-source LLM that crosses a transformer with an SSM原评论 ↗
被提交为 story · 1 次