按年被提到
20212023 · 2 次2026
大模型与语言模型›模型表达力与计算理论·arXiv:2106.16213 · cs.CL

Saturated Transformers are Constant-Depth Threshold Circuits

William Merrill, Ashish Sabharwal, Noah A. Smith · 2021
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-04-18 · 最近 2023-06-17
2不同的人
2活跃月份
2讨论串
25被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Transformers have become a standard neural network architecture for many NLP problems, motivating theoretical analysis of their power in terms of formal languages. Recent work has shown that transformers with hard attention are quite limited in power (Hahn, 2020), as they can be simulated by constant-depth AND/OR circuits (Hao et al. 2021). However, hard attention is a strong assumption, which may complicate the relevance of these results in practice. In this work, we analyze the circuit complexity of transformers with saturated attention: a generalization of hard attention that more closely captures the attention patterns learnable in practical transformers. We first show that saturated transformers transcend the known limitations of hard-attention transformers. We then prove saturated transformers with floating-point values can be simulated by constant-depth threshold circuits, giving the class $\mathsf{TC}^0$ as an upper bound on the formal languages they recognize.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论shawntan · 2023-04

“There's more detailed examination of Transformers from circuit complexity in this paper: https://arxiv.org/abs/2106.16213 There are a few references also in the blog post.”

The New XOR Problem原评论 ↗
评论bmc7505 · 2023-06

“The current generation of LLMs will never generalize on problems whose expected complexity is significantly harder than the training set. One way to think of why this is the case is because transformers are learning shortcuts to multistep reasoning tasks [1, 2], but do not learn to reason systematically and unsurprisingly fail to generalize when presented…”

Researchers warn of ‘model collapse’ as AI trains on AI-generated content原评论 ↗