按年被提到
20242024 · 1 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:2410.06205 · cs.CL

Round and Round We Go! What makes Rotary Positional Encodings useful?

Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, Petar Veličković · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-11-18 · 最近 2026-09-10
2不同的人
2活跃月份
2讨论串
9被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings (RoPE), that rotate the queries and keys based on their relative distance. A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to use RoPE to construct robust "positional" attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to use the lowest frequencies of RoPE, which we suspect are used to carry semantic information. We mathematically prove interesting behaviours of RoPE and conduct experiments to verify our findings, proposing a modification of RoPE that fixes some highlighted issues and improves performance. We believe that this work represents an interesting step in better understanding PEs in LLMs, which we believe holds crucial value for scaling LLMs to large sizes and context lengths.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

没有评论直接链到它;下面是它被提交的讨论串里回复最多的评论(不计入统计)
讨论串评论fuzzfactor · 2026-09

“What do people do when they don't know when to stop, and what is the digital/simulated/artificial equivalent anyway? In electronics, rotary positional encoders are often used to replace traditional analog potentiometers (variable resistors) for things like audio volume controls or frequency equalizers. Like the title says "Round and Round We Go!" this e…”

What Makes Rotary Positional Encodings Useful?原评论 ↗
被提交为 story · 2 次