“It turns out that transformers have a learning mechanism similar to autodiff but better since it happens mostly within the single layers as opposed to over the whole graph. I wrote a paper on this recently https://arxiv.org/abs/2302.01834v1. The math is crazy.”
Coinductive guide to inductive transformer heads
We argue that all building blocks of transformer models can be expressed with a single concept: combinatorial Hopf algebra. Transformer learning emerges as a result of the subtle interplay between the algebraic and coalgebraic operations of the combinatorial Hopf algebra. Viewed through this lens, the transformer model becomes a linear time-invariant system where the attention mechanism computes a generalized convolution transform and the residual stream serves as a unit impulse. Attention-only transformers then learn by enforcing an invariant between these two paths. We call this invariant Hopf coherence. Due to this, with a degree of poetic license, one could call combinatorial Hopf algebras "tensors with a built-in loss function gradient". This loss function gradient occurs within the single layers and no backward pass is needed. This is in contrast to automatic differentiation which happens across the whole graph and needs a explicit backward pass. This property is the result of the fact that combinatorial Hopf algebras have the surprising property of calculating eigenvalues by repeated squaring.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834v1 The learning mechanism of transformer models was poorly understood however it turns out that a transformer is like a circuit with a feedback. I argue that autodiff can be replaced with what I call in the p…”
“I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834 The learning mechanism of transformer models was poorly understood however it turns out that a transformer is like a circuit with a feedback. I argue that autodiff can be replaced with what I call in the pape…”
“I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834 This ties really nicely to the photonic DSP as convolution is fundamentally composition of convolutive systems. This is a generalized convolution, not the standard one. The learning mechanism of transformer m…”
“I have recently written a paper on understanding machine learning via the lens of Hopf algebra https://arxiv.org/abs/2302.01834. Hopf algebras (which are really just tensors with recurrence relations built in) subsume convnets, transformers and diffusion model and also provide a theoretically better autodiff that operates within single layers as opposed …”
“I have recently written a paper on understanding machine learning via the lens of Hopf algebra https://arxiv.org/abs/2302.01834. Hopf algebras (which are really just tensors with recurrence relations built in) subsume convnets, transformers and diffusion model and also provide a theoretically better autodiff that operates within single layers as opposed …”
看全部 16 条 →
“Intersting papers. https://arxiv.org/abs/2302.01834 appears to have a typo in section 4.5 S(hg) = S(g)S(g) looks like it should be S(hg) = S(h)S(g) or S(hg) = S(g)S(h)”
“I'm pretty sure that all of ML is subsumed by Hopf algebra. I have written a paper on this recently https://arxiv.org/abs/2302.01834 I also have a discord channel https://discord.cofunctional.ai.”
“I have written a paper on this recently viewing transformers as Hopf algebras https://arxiv.org/abs/2302.01834. This paper is very relevant https://arxiv.org/abs/1206.3620.”
“Hopf algebra subsumes like all all currently fashionable architectures (transformers, convnets, SSM, diffusion models) and then some. Hopf convolution is a surprisingly general concept, Groebner bases are the thing used to calculate the antipode. Hopf algebra is tensor bialgebra, i.e. tensor with a built in feedback (which subsumes autodiff). I have wri…”
“The attention mechanism corresponds to the Hopf algebraic convolution, a generalization of the commonly known convolution. I'm in the process of implementing a framework based on this idea. I have written a paper on this recently, https://arxiv.org/abs/2302.01834 I have a discord channel https://discord.cofunctional.ai.”
“I argue that the attention mechanism is just Hopf algebra convolution. The QK mechanism selects possible candidates and the OV mechanism picks one of these possible candidates. I have recently written a paper on this https://arxiv.org/abs/2302.01834 I also have a discord channel https://discord.cofunctional.ai. Ironically, it's the same mechanism as wh…”
“I have figured out a way of doing machine learning right. Turns out that all popular architectures are just instances of a more general approach. Infinite transformer window is possible. Attention is just a particular case of Hopf convolution. Verification is possible. Faster training, faster inference. Possibly interpretability. But it goes beyond transf…”
“I have figured out a way of doing machine learning right. Turns out that all popular architectures are just instances of a more general approach. Infinite transformer window is possible. Attention is just a particular case of Hopf convolution. Verification is possible. Faster training, faster inference. Possibly interpretability. But it goes beyond transf…”
“I have figured out a way of doing machine learning right. Turns out that all popular architectures are just instances of a more general approach. Infinite transformer window is possible. Attention is just a particular case of Hopf convolution. Verification is possible. Faster training, faster inference. Possibly interpretability. But it goes beyond transf…”
“I have figured out a way of doing machine learning right. Turns out that all popular architectures are just instances of a more general approach. Infinite transformer window is possible. Attention is just a particular case of Hopf convolution. Verification is possible. Faster training, faster inference. Possibly interpretability. But it goes beyond transf…”
- Coinductive guide to inductive transformer heads ↗adamnemecek · 2023-02-06 · 4 分
- Coinductive guide to inductive transformer heads ↗adamnemecek · 2023-02-13 · 1 分
- Coinductive guide to inductive transformer heads ↗adamnemecek · 2023-02-14 · 1 分
- Transformer learning explained: Coinductive guide to inductive transformer heads ↗adamnemecek · 2023-02-24 · 2 分