“It's a couple years old but the best I've seen in this area is Retrieval Augmented Generation [2].”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
RAG:把检索到的文档与生成模型结合
论文提出检索增强生成(RAG):以预训练的 seq2seq 模型作为参数化记忆,以维基百科的稠密向量索引作为非参数化记忆,并通过预训练的神经检索器访问。作者比较了两种形式,一种在整个生成序列中使用同一批检索段落,另一种每个 token 可用不同段落;在三个开放域问答任务上刷新最好成绩,在语言生成任务上也比纯参数化 seq2seq 基线更具体、多样、符合事实。
评论者把它当作「RAG」这个名称的原始出处来引用,在讨论长上下文的替代方案和生成式搜索时用它说明「检索加生成」的模式。
学习路径
读这篇之前,按顺序读书架上的 7 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出先补 Transformer、BERT、T5 这条预训练与序列到序列的主线,再看外部记忆、向量检索索引和 REALM 的检索增强预训练,就能读懂 RAG 把检索器和生成器接在一起的做法。
- 1
- 2
RAG 的查询和文档编码器是 BERT 式的双向编码器。
读预训练加微调的范式,以及 [CLS] 表示怎么用;跳过各任务细节。
- 3
RAG 的生成端是文本到文本的序列到序列模型,T5 展示了这种统一形式。
读文本到文本框架的思路,以及编码器-解码器预训练目标的概念。
- 4
- 5
向量检索靠近似最近邻索引,HNSW 是常用的一类图索引。
读多层邻近图与对数复杂度搜索的基本思路;不必深究建图参数。
- 6
RAG 的文档向量库用 FAISS 做最大内积搜索,这篇是其核心实现依据。
读乘积量化和 GPU 上 k-选择的概念,弄清为什么能支持亿级文档检索。
- 7
REALM 把可学习的检索器放进语言模型预训练,是 RAG 最直接的同类前作,要对比两者怎么处理检索。
读检索器与语言模型联合训练的做法,以及检索结果如何作为潜变量被边缘化。
- 8Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
目标论文:把预训练检索器与序列到序列生成器端到端结合,用于知识密集型任务。
读RAG-Sequence 与 RAG-Token 两种边缘化方式,以及检索器与生成器的联合微调;对照 REALM 看区别。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“There was a 2020 paper called Retrieval Augmented Generation that describes this technique too [1].”
“This more or less is already a thing and it's called RAG [1][2].”
“then using some type of Retrieval Augmented Generation (RAG) to process that using the extended contexts ?”
“Isn’t Retrieval-Augmented Generation the standard nomenclature for this pattern?”
“RAG has been studied in various research papers, such as the one available at Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”
看全部 14 条 →
“this concept already exists and is in practice at many companies that require knowledge driven results.”
“The paper that coined the term used the hyphen, though I think I prefer it without: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”
“retrieval augmented generation - Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”
“I'm not sure this is the paper that coined it, but here's an example from 2020: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”
“Original RAG paper: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”
“We all know that Generative Search Engines are built on RAG (Retrieval-Augmented Generation)[1] combined with Large Language Models (LLMs).”
“We all know that Generative Search Engines are built on RAG (Retrieval-Augmented Generation)[1] combined with Large Language Models (LLMs).”
“Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2021) (apparently the original paper where the term 'RAG' was coined):”
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks ↗cl42 · 2021-03-14 · 1 分
- RAG: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks ↗teleforce · 2023-12-01 · 2 分