按年被提到
20232023 · 22 次2026
大模型与语言模型›模型表达力与计算理论·arXiv:2302.01834 · cs.LG

Coinductive guide to inductive transformer heads

Adam Nemecek · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-02-06 · 最近 2023-09-01
2不同的人
8活跃月份
21讨论串
0被引(OpenAlex)
arXiv 摘要(候选层不写导读)

We argue that all building blocks of transformer models can be expressed with a single concept: combinatorial Hopf algebra. Transformer learning emerges as a result of the subtle interplay between the algebraic and coalgebraic operations of the combinatorial Hopf algebra. Viewed through this lens, the transformer model becomes a linear time-invariant system where the attention mechanism computes a generalized convolution transform and the residual stream serves as a unit impulse. Attention-only transformers then learn by enforcing an invariant between these two paths. We call this invariant Hopf coherence. Due to this, with a degree of poetic license, one could call combinatorial Hopf algebras "tensors with a built-in loss function gradient". This loss function gradient occurs within the single layers and no backward pass is needed. This is in contrast to automatic differentiation which happens across the whole graph and needs a explicit backward pass. This property is the result of the fact that combinatorial Hopf algebras have the surprising property of calculating eigenvalues by repeated squaring.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论adamnemecek · 2023-02

“It turns out that transformers have a learning mechanism similar to autodiff but better since it happens mostly within the single layers as opposed to over the whole graph. I wrote a paper on this recently https://arxiv.org/abs/2302.01834v1. The math is crazy.”

GPT in 60 Lines of NumPy原评论 ↗
评论adamnemecek · 2023-02

“I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834v1 The learning mechanism of transformer models was poorly understood however it turns out that a transformer is like a circuit with a feedback. I argue that autodiff can be replaced with what I call in the p…”

Transformers Catalog: 60 Transformers in 36 pages原评论 ↗
评论adamnemecek · 2023-02

“I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834 The learning mechanism of transformer models was poorly understood however it turns out that a transformer is like a circuit with a feedback. I argue that autodiff can be replaced with what I call in the pape…”

Running large language models like ChatGPT on a single GPU原评论 ↗
评论adamnemecek · 2023-02

“I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834 This ties really nicely to the photonic DSP as convolution is fundamentally composition of convolutive systems. This is a generalized convolution, not the standard one. The learning mechanism of transformer m…”

Optical Transformers原评论 ↗
评论adamnemecek · 2023-03

“I have recently written a paper on understanding machine learning via the lens of Hopf algebra https://arxiv.org/abs/2302.01834. Hopf algebras (which are really just tensors with recurrence relations built in) subsume convnets, transformers and diffusion model and also provide a theoretically better autodiff that operates within single layers as opposed …”

7 Weeks AI Alignment Curriculum原评论 ↗
评论adamnemecek · 2023-03

“I have recently written a paper on understanding machine learning via the lens of Hopf algebra https://arxiv.org/abs/2302.01834. Hopf algebras (which are really just tensors with recurrence relations built in) subsume convnets, transformers and diffusion model and also provide a theoretically better autodiff that operates within single layers as opposed …”

Category Theory ∩ Machine Learning原评论 ↗
看全部 16 条 →
评论nicwilson · 2023-03

“Intersting papers. https://arxiv.org/abs/2302.01834 appears to have a typo in section 4.5 S(hg) = S(g)S(g) looks like it should be S(hg) = S(h)S(g) or S(hg) = S(g)S(h)”

Category Theory ∩ Machine Learning原评论 ↗
评论adamnemecek · 2023-04

“I'm pretty sure that all of ML is subsumed by Hopf algebra. I have written a paper on this recently https://arxiv.org/abs/2302.01834 I also have a discord channel https://discord.cofunctional.ai.”

Neural Networks: Zero to Hero原评论 ↗
评论adamnemecek · 2023-04

“I have written a paper on this recently viewing transformers as Hopf algebras https://arxiv.org/abs/2302.01834. This paper is very relevant https://arxiv.org/abs/1206.3620.”

A Baby GPT原评论 ↗
评论adamnemecek · 2023-04

“Hopf algebra subsumes like all all currently fashionable architectures (transformers, convnets, SSM, diffusion models) and then some. Hopf convolution is a surprisingly general concept, Groebner bases are the thing used to calculate the antipode. Hopf algebra is tensor bialgebra, i.e. tensor with a built in feedback (which subsumes autodiff). I have wri…”

Use Gröbner Bases to Solve Polynomial Equations原评论 ↗
评论adamnemecek · 2023-05

“The attention mechanism corresponds to the Hopf algebraic convolution, a generalization of the commonly known convolution. I'm in the process of implementing a framework based on this idea. I have written a paper on this recently, https://arxiv.org/abs/2302.01834 I have a discord channel https://discord.cofunctional.ai.”

Unlimiformer: Long-Range Transformers with Unlimited Length Input原评论 ↗
评论adamnemecek · 2023-05

“I argue that the attention mechanism is just Hopf algebra convolution. The QK mechanism selects possible candidates and the OV mechanism picks one of these possible candidates. I have recently written a paper on this https://arxiv.org/abs/2302.01834 I also have a discord channel https://discord.cofunctional.ai. Ironically, it's the same mechanism as wh…”

Ask HN: Can someone ELI5 Transformers and the “Attention is all we need” paper原评论 ↗
评论adamnemecek · 2023-06

“I have figured out a way of doing machine learning right. Turns out that all popular architectures are just instances of a more general approach. Infinite transformer window is possible. Attention is just a particular case of Hopf convolution. Verification is possible. Faster training, faster inference. Possibly interpretability. But it goes beyond transf…”

Ask HN: Who wants to be hired? (June 2023)原评论 ↗
评论adamnemecek · 2023-07

“I have figured out a way of doing machine learning right. Turns out that all popular architectures are just instances of a more general approach. Infinite transformer window is possible. Attention is just a particular case of Hopf convolution. Verification is possible. Faster training, faster inference. Possibly interpretability. But it goes beyond transf…”

Ask HN: Who wants to be hired? (July 2023)原评论 ↗
评论adamnemecek · 2023-08

“I have figured out a way of doing machine learning right. Turns out that all popular architectures are just instances of a more general approach. Infinite transformer window is possible. Attention is just a particular case of Hopf convolution. Verification is possible. Faster training, faster inference. Possibly interpretability. But it goes beyond transf…”

Ask HN: Who wants to be hired? (August 2023)原评论 ↗
评论adamnemecek · 2023-09

“I have figured out a way of doing machine learning right. Turns out that all popular architectures are just instances of a more general approach. Infinite transformer window is possible. Attention is just a particular case of Hopf convolution. Verification is possible. Faster training, faster inference. Possibly interpretability. But it goes beyond transf…”

Ask HN: Who wants to be hired? (September 2023)原评论 ↗
被提交为 story · 4 次