按年被提到
20242024 · 1 次2026
大模型与语言模型›检索增强与知识库·arXiv:2403.10131 · cs.CL

RAFT: Adapting Language Model to Domain Specific RAG

Tianjun Zhang, Shishir G. Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, Joseph E. Gonzalez · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-03-18 · 最近 2026-10-28
2不同的人
2活跃月份
2讨论串
150被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Pretraining Large Language Models (LLMs) on large corpora of textual data is now a standard paradigm. When using these LLMs for many downstream applications, it is common to additionally bake in new knowledge (e.g., time-critical news, or private domain knowledge) into the pretrained model either through RAG-based-prompting, or fine-tuning. However, the optimal methodology for the model to gain such new knowledge remains an open question. In this paper, we present Retrieval Augmented FineTuning (RAFT), a training recipe that improves the model's ability to answer questions in a "open-book" in-domain settings. In RAFT, given a question, and a set of retrieved documents, we train the model to ignore those documents that don't help in answering the question, which we call, distractor documents. RAFT accomplishes this by citing verbatim the right sequence from the relevant document that would help answer the question. This coupled with RAFT's chain-of-thought-style response helps improve the model's ability to reason. In domain-specific RAG, RAFT consistently improves the model's performance across PubMed, HotpotQA, and Gorilla datasets, presenting a post-training recipe to improve pre-trained LLMs to in-domain RAG. RAFT's code and demo are open-sourced at github.com/ShishirPatil/gorilla.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论gopalv · 2026-10

“> On-policy distillation: The expanded student model learns token-specific feedback from teacher distributions generated on self-sampled sequences. The weird thing is to see this based off the KV-Cache in the diagrams. Completely different from the Databricks RAFT [1]. My team built "Dreamtime" earlier this year [2], which used a different mechanism to…”

Sleep framework: A new way for LLMs to retain knowledge in continual learning原评论 ↗
被提交为 story · 1 次