引用thatguysaguy · 2024-02
“They cannot do arbitrary key-based recall from their context: Repeat After Me”
引用论文说明 SSM 无法按任意键从上下文中回忆内容
证明固定大小隐状态的模型在上下文复制与检索上不如 Transformer
论文研究“广义状态空间模型”(GSSM,隐状态大小固定)在复制输入上下文这类任务上的能力。理论上证明两层 Transformer 可以复制指数长度的字符串,而 GSSM 受固定隐状态限制;在合成任务和预训练大模型上的实验也显示 Transformer 在复制与检索上明显优于状态空间模型。
评论者用它说明没有注意力的模型(如 Mamba、线性注意力)在上下文回忆类任务上仍落后于 softmax 注意力。
“They cannot do arbitrary key-based recall from their context: Repeat After Me”
“Indeed: Repeat After Me”
“Nothing has outperformed attention so far, not even Mamba: Repeat After Me”
“Models with attention do well on recall tasks, but models without it, like this one, tend to do poorly.”
“My understanding and experience is that all the linear attention models out there [b] still underperform Softmax attention on things like recall tasks.[c]”
“Relevant: Repeat After Me”