“Very cool work.”
Hyena Hierarchy: Towards Larger Convolutional Language Models
用长卷积加门控替代注意力的亚二次复杂度算子
论文提出 Hyena,一种亚二次复杂度的注意力替代算子,由隐式参数化的长卷积与数据控制的门控交替构成。在数千到数十万长度序列的召回和推理任务上,准确率比基于状态空间等算子的方法高出 50 多点,与基于注意力的模型持平;在 WikiText103 和 The Pile 上刷新了无稠密注意力架构的最好成绩,序列长 2K 时训练算力减少 20% 即达到 Transformer 质量,并在 8K 长度下比优化后的注意力快两倍、64K 下快 100 倍。
它常被当作「注意力的长上下文替代方案」之一提及,评论者借它讨论上下文长度的扩展;也有评论批评后来的工作没有引用 Hyena。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“I see no mention of prior work Hyena Operator which already demonstrated O(n log n) full context mixing several years ago.”
“"Hyena Hierarchy: Towards Larger Convolutional Language Models" Hyena Hierarchy”
“Rumor is RedPajama is going to have upwards of 60k token context by using Hyena: Hyena Hierarchy.”
“Title of popsci is overreaching, this is a drop-in subquadratic replacement for attention.”
“Has Hyena not addressed the scaling of context length [1]?”
看全部 7 条 →
“And the stanford heyena paper: Hyena Hierarchy”
- Hyena Hierarchy: Towards Larger Convolutional Language Models ↗bookofjoe · 2023-03-10 · 1 分
- Hyena Hierarchy: Towards Larger Convolutional Language Models ↗beefman · 2023-04-02 · 2 分
- Hyena a more efficient convolutional LLM when compared to GPT ↗maxpert · 2023-04-21 · 2 分
- Hyena Hierarchy: Towards Larger Convolutional Language Models ↗quantisan · 2023-04-21 · 1 分