推荐nextos · 2025-02
“The xLSTM could become a good alternative to transformers: xLSTM. On very long contexts, such as those arising in DNA models, these models perform really well.”
推荐 xLSTM 作为 Transformer 的好替代,在 DNA 模型这类超长上下文中表现很好
给 LSTM 加指数门控与矩阵记忆,规模化后可比肩 Transformer
探讨把 LSTM 扩展到数十亿参数、结合现代 LLM 技术并缓解其已知局限能走多远。做法是引入带归一化与稳定化的指数门控,并改造记忆结构:sLSTM(标量记忆、标量更新、新的记忆混合)和可完全并行的 mLSTM(矩阵记忆与协方差更新规则),再把它们放进残差块堆叠成 xLSTM。在性能与规模扩展上与最先进的 Transformer 和状态空间模型相比表现良好。
HN 上的人把它视为 Transformer 的可能替代,在超长上下文(如 DNA 模型)表现好、解决了状态跟踪难题,但也有人认为其质量/参数比并未明显超过 Transformer。
“The xLSTM could become a good alternative to transformers: xLSTM. On very long contexts, such as those arising in DNA models, these models perform really well.”
“It does seem to match transformers, but I wouldn't say it meaningfully outperforms them in terms of quality vs parameters.”
“which demonstrates a significant improvement on the kind of state-tracking problems that transformers struggle with.”
“[10] xLSTM”