“you should check out the new Mamba (Transformers replacement) architecture: Mamba”
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
选择性状态空间模型:线性复杂度的 Transformer 替代
论文指出结构化状态空间模型(SSM)的关键弱点是无法做基于内容的推理,于是让 SSM 的参数成为输入的函数,使模型能按当前 token 选择性地沿序列传播或遗忘信息,并设计了适配硬件的并行循环算法来弥补由此失去的卷积加速。由此得到不含注意力甚至 MLP 块的 Mamba 架构,推理吞吐量约为 Transformer 的 5 倍,序列长度上线性扩展;Mamba-3B 在语言建模上优于同尺寸 Transformer,并可比肩两倍大小的 Transformer。
评论者主要把它当作 Transformer 的线性复杂度替代架构来引用,用来讨论长上下文,询问某个新模型是否用了 Mamba,或引用其缩放定律图。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“This is a reddit post which links to a Twitter post [0] which links to this paper [1] and this repo [2]”
“which does explain what Mamba is by linking to the original paper”
“Mamba: Linear-Time Sequence Modeling with Selective State Spaces Mamba”
“Mamba did such benchmark.”
“they are using the new SAMBA architecture.”
看全部 16 条 →
“For more info, see the scaling laws plot in Figure 4 of the Mamba paper: Mamba”
“Yes you're spot on, the nonlinearities come from the full Mamba blocks”
“Does anyone know if this is using the Mamba architecture[1] instead of transformers?”
“A paper describing how you might do it published in December last year.”
“It's an LLM architecture competing with transformers: Mamba”
“has an equivalent unit to a gated RNN”
“[6] Mamba”
“E.g.: Mamba”
“We mentioned Mamba, they compared to GPT-3 in their first paper[0].”
“* State space models: Mamba / [OTHER_PAPER]”
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces ↗kvee · 2023-12-04 · 1 分
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces ↗anigbrowl · 2023-12-04 · 2 分