按年被提到
20262026 · 5 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:2607.03502 · cs.CL

Reading Between the Dots: Decoding Hidden Computation across Filler Tokens

Kaley Brauer, Claudio Mayrink Verdun, Samuel Marks · 2026
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 5 人首次 2026-07-18 · 最近 2026-08-14
5不同的人
2活跃月份
5讨论串
0被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Frontier LLMs can perform multi-step reasoning over content-free filler tokens like dots or counting sequences, producing correct answers with no visible chain-of-thought (CoT). This is a limit case for behavioral oversight, where surface tokens carry no information about the underlying reasoning. But hidden from the output is not the same as hidden from us. On four task families (fact retrieval, parallel numeric composition, string manipulation, and in-context computation), two open-weights frontier models (DeepSeek V3, Kimi K2) compute over filler tokens in a legible way: attention routes the question through the filler region to the answer, logit-lens readouts show retrieved facts emerging early and their composition crystallizing in late layers, and KV-cache transplants at filler positions causally swap outputs between examples. We introduce an unsupervised decoding pipeline that takes only hidden states as input and recovers intermediate values with 82-94% accuracy (best LLM judge) across both models and all four tasks, without ground-truth labels or training. Even without a judge, the hidden values are already directly in the pipeline's top-2 tokens 35-85% of the time. The uplift persists whether the filler is prefilled or the model generates the filler itself. On these cleanly decomposable tasks, hidden computation that defeats behavioral CoT monitoring is readable from the residual stream, which suggests that monitorability is a property of the model's full computational trace rather than only its surface tokens.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论dnautics · 2026-08

“> to conclude if it is thinking or not I would want to look inside its head https://arxiv.org/abs/2607.03502 a non-thinking token model (just "completion") can answer one-step questions but generally not multistep questions. however, if you append [n] of a single token (e.g. period, space), it is able to use the activations in the higher layers of the b…”

Why does Opus 5 feel worse to work with?原评论 ↗
被提交为 story · 4 次