按年被提到
20242024 · 1 次2026
大模型与语言模型›模型表达力与计算理论·arXiv:2402.08164 · stat.ML

On Limitations of the Transformer Architecture

Binghui Peng, Srini Narayanan, Christos Papadimitriou · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-04-08 · 最近 2025-04-21
2不同的人
2活跃月份
2讨论串
23被引(OpenAlex)
arXiv 摘要(候选层不写导读)

What are the root causes of hallucinations in large language models (LLMs)? We use Communication Complexity to prove that the Transformer layer is incapable of composing functions (e.g., identify a grandparent of a person in a genealogy) if the domains of the functions are large enough; we show through examples that this inability is already empirically present when the domains are quite small. We also point out that several mathematical tasks that are at the core of the so-called compositional tasks thought to be hard for LLMs are unlikely to be solvable by Transformers, for large enough instances and assuming that certain well accepted conjectures in the field of Computational Complexity are true.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论JohnKemeny · 2024-04

“There's a paper by Papadimitriou (from Logicomix fame) and some collaborators that the transformer model is incapable of solving certain simple problems, and if done by Cost, it needs exponentially many steps. The paper is currently only available at Arxiv (ie not yet peer-reviewed), but given that it is Papadimitriou, I would be inclined to believe the …”

Ask HN: Is anybody getting value from AI Agents? How so?原评论 ↗
评论namaria · 2025-04

“Theoretical limitations of multi-layer Transformer https://arxiv.org/abs/2412.02975 On Limitations of the Transformer Architecture https://arxiv.org/abs/2402.08164 Limits of Deep Learning: Sequence Modeling through the Lens of Complexity Theory https://arxiv.org/abs/2405.16674 TL;DR transformers are inherently limited with tasks requiring composition o…”

LLM-powered tools amplify developer capabilities rather than replacing them原评论 ↗