按年被提到
20222023 · 2 次2026
大模型与语言模型›模型表达力与计算理论·arXiv:2210.02671 · cs.LG

A Logic for Expressing Log-Precision Transformers

William Merrill, Ashish Sabharwal · 2022
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-04-25 · 最近 2023-05-26
2不同的人
2活跃月份
2讨论串
8被引(OpenAlex)
arXiv 摘要(候选层不写导读)

One way to interpret the reasoning power of transformer-based language models is to describe the types of logical rules they can resolve over some input text. Recently, Chiang et al. (2023) showed that finite-precision transformers can be equivalently expressed in a generalization of first-order logic. However, finite-precision transformers are a weak transformer variant because, as we show, a single head can only attend to a constant number of tokens and, in particular, cannot represent uniform attention. Since attending broadly is a core capability for transformers, we ask whether a minimally more expressive model that can attend universally can also be characterized in logic. To this end, we analyze transformers whose forward pass is computed in $\log n$ precision on contexts of length $n$. We prove that any log-precision transformer can be equivalently expressed as a first-order logic sentence that, in addition to standard universal and existential quantifiers, may also contain majority-vote quantifiers. This is the tightest known upper bound and first logical characterization of log-precision transformers.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论bmc7505 · 2023-04

“See Merrill & Sabharwal (2023) [1]: A transformer N is a specific kind of a computation graph family over vectors of floating-point numbers with p precision. We work with fixed-precision transformers, i.e., we take p to be constant with respect to the input sequence length n. In practice, transformers typically use p = 32, and, several newer, larger tran…”

Tighter bounds on the expressivity of transformer encoders原评论 ↗
被提交为 story · 1 次