按年被提到
20232024 · 1 次2026
大模型与语言模型›大模型预训练与缩放·arXiv:2305.18466 · cs.CL

Test-Time Training on Nearest Neighbors for Large Language Models

Moritz Hardt, Yu Sun · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-11-18 · 最近 2025-08-02
2不同的人
2活跃月份
2讨论串
6被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Many recent efforts augment language models with retrieval, by adding retrieved data to the input context. For this approach to succeed, the retrieved data must be added at both training and test time. Moreover, as input length grows linearly with the size of retrieved data, cost in computation and memory grows quadratically for modern Transformers. To avoid these complications, we simply fine-tune the model on retrieved data at test time, using its standard training setup. We build a large-scale distributed index based on text embeddings of the Pile dataset. For each test input, our system retrieves its neighbors and fine-tunes the model on their text. Surprisingly, retrieving and training on as few as 20 neighbors, each for only one gradient iteration, drastically improves performance across more than 20 language modeling tasks in the Pile. For example, test-time training with nearest neighbors significantly narrows the performance gap between a small GPT-2 and a GPT-Neo model more than 10 times larger. Sufficient index quality and size, however, are necessary. Our work establishes a first baseline of test-time training for language modeling.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论qrios · 2025-08

“Current "good enough" models like Mistral Small require GPUs like the RTX 6000 to achieve user-friendly response times. The model quality is good enough, especially for narrow-scope tasks like summarization and classification. If Moore's Law holds for a few more years, a mobile device will be able to run it on-device in around 8 years (Apple's A11: 410 GF…”

Tim Cook rallying Apple employees around AI efforts原评论 ↗
被提交为 story · 1 次