按年被提到
20242025 · 2 次2026
大模型与语言模型›模型表达力与计算理论·arXiv:2406.03445 · cs.LG

Pre-trained Large Language Models Use Fourier Features to Compute Addition

Tianyi Zhou, Deqing Fu, Vatsal Sharan, Robin Jia · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2025-02-06 · 最近 2025-08-22
2不同的人
2活跃月份
2讨论串
4被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Pre-trained large language models (LLMs) exhibit impressive mathematical reasoning capabilities, yet how they compute basic arithmetic, such as addition, remains unclear. This paper shows that pre-trained LLMs add numbers using Fourier features -- dimensions in the hidden state that represent numbers via a set of features sparse in the frequency domain. Within the model, MLP and attention layers use Fourier features in complementary ways: MLP layers primarily approximate the magnitude of the answer using low-frequency features, while attention layers primarily perform modular addition (e.g., computing whether the answer is even or odd) using high-frequency features. Pre-training is crucial for this mechanism: models trained from scratch to add numbers only exploit low-frequency features, leading to lower accuracy. Introducing pre-trained token embeddings to a randomly initialized model rescues its performance. Overall, our analysis demonstrates that appropriate pre-trained representations (e.g., Fourier features) can unlock the ability of Transformers to learn precise mechanisms for algorithmic tasks.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论astrobe_ · 2025-08

“"Isn't it just temporary, and AI will get better?" There are 2 instances of the word "understand" in the first paragraph, 3 if you count the beginning of the second. In my book, "understanding" is a synonym for "intelligence" - the roots of the word are "read between the lines", where something else that just knowledge is, the ability to use and manipul…”

Vibe Debugging: Enterprises' Up and Coming Nightmare原评论 ↗
被提交为 story · 1 次