按年被提到
20212022 · 7 次2026
大模型与语言模型›量化、解码与分词·arXiv:2112.04426 · cs.CL

Improving language models by retrieving from trillions of tokens

通过从数万亿 token 中检索来改进语言模型
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy 等 28 人 · 2021

RETRO:用万亿级 token 检索库增强自回归语言模型

开始阅读 ↗PDFalphaXiv 讨论
被提及 · 8 人首次 2022-01-24 · 最近 2024-11-10
8不同的人
8活跃月份
12讨论串
490被引(OpenAlex)

论文让自回归语言模型以此前 token 的局部相似度从大语料库中检索文档块,并以此为条件生成,提出 Retrieval-Enhanced Transformer(RETRO)。在 2 万亿 token 的数据库下,RETRO 在 Pile 上以少 25 倍的参数取得与 GPT-3 和 Jurassic-1 相当的表现;它结合冻结的 BERT 检索器、可微编码器和分块交叉注意力,可从头训练,也可对已有 Transformer 做 RETROfit,微调后还能用于问答等知识密集任务。

HN 为什么提它

评论者把它当作“检索增强、外部记忆让小模型媲美大模型”的代表,也拿它讨论训练集与测试集的数据泄漏计算和 MassiveText 语料。

方法难度 4 / 5约 80 分钟经得起时间
读之前最好知道
1Transformer 架构
2交叉注意力(cross-attention)
3语言模型评估(bits-per-byte)
4最近邻检索与向量检索

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
引用visarga · 2022-01

“The trend now is to make them multi language, multi modal, multi task, multi memory ([1] and this paper)”

把它与另一篇并举,说明语言模型正走向多语言、多模态、多任务、多记忆
WebGPT: Browser-assisted question-answering with human feedback原评论 ↗
引用ahgamut · 2022-02

“In recent literature, the RETRO Transformer talks about "quantifying dataset leakage", which is related to what I mentioned in the above paragraph.”

引用 RETRO 对量化数据集泄漏的讨论,呼应自己前文所说
Competitive Programming with AlphaCode原评论 ↗
引用ahgamut · 2022-02

“and the RETRO transformer has a calculation for the effect of "dataset leakage".”

指出 RETRO 计算了数据集泄漏的影响
AlphaCode as a dog speaking mediocre English原评论 ↗
引用sanxiyn · 2022-07

“If you are talking about Figure 1 of Improving language models by retrieving from trillions of tokens, you are reading it wrong. Y axis is bits-per-byte and lower is better.”

纠正对 Figure 1 的误读:纵轴是 bits-per-byte,越低越好
RETRO is fast原评论 ↗
引用lossolo · 2022-12

“11B parameters, so I guess it's FLAN-T5[1]? combined with retrieval transformer (something like RETRO[2])?”

猜测该搜索系统可能是 FLAN-T5 加上类似 RETRO 的检索 Transformer
Show HN: Web search using a ChatGPT-like model that can cite its sources原评论 ↗
引用lossolo · 2022-12

“or Transformers paper [2]?”

把它列入 Google 发布的 NLP 模型之中,未作评价
AlphaCode Attention Visualization原评论 ↗
看全部 12 条 →
引用lossolo · 2022-12

“Here is example of one from Google: RETRO[1][2]”

以 Google 的 RETRO 作为检索式模型的例子
How Does GPT Obtain Its Ability? Tracing Emergent Abilities of Language Models原评论 ↗
引用antimatter15 · 2023-01

“I would be excited about an open-source RETRO model with comparable performance to GPT-3”

期待有性能媲美 GPT-3 的开源 RETRO 模型
Anthropic's Claude is said to improve on ChatGPT, but still has limitations原评论 ↗
引用halflings · 2023-01

“Examples of such architectures:”

把 RETRO 列为外部记忆架构的例子
Neural Networks and the Chomsky Hierarchy [DeepMind, 2022]原评论 ↗
引用sanxiyn · 2023-05

“DeepMind's RETRO paper Improving language models by retrieving from trillions of tokens mentions a dataset called MassiveText, which includes 20 million books of 3T tokens.”

引用 RETRO 论文提到的 MassiveText 数据集,含 2000 万本书共 3T token
PaLM 2 Technical Report [pdf]原评论 ↗
引用magicalhippo · 2024-06

“From what I gather the cross-attention allows the expert selection to react to the context, unlike the more naive plain Softmax gating approach.”

解释 cross-attention 让专家选择能随上下文变化,不同于朴素的 Softmax 门控
Lamini Memory Tuning: 10x Fewer Hallucinations原评论 ↗
引用williamtrask · 2024-11

“This paper from DeepMind a few years ago offers a counter example to this claim.”

拿 DeepMind 这篇作反例,反驳前面的说法:检索能让小模型与大模型竞争
LLMs have reached a point of diminishing returns原评论 ↗