“For anyone interested in these LLM pairwise sorting problems, check out this paper: Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting”
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting
成对排序提示 PRP:让中等规模开源 LLM 达到文本排序最优
作者指出现有逐点和列表式排序提示对现成 LLM 过于困难,提出成对排序提示(PRP),只让模型比较两篇文档。在 TREC-DL 2019/2020 上,20B 的 Flan-UL2 配合 PRP 与此前基于 GPT-4 的最佳方法相当,在七个 BEIR 任务上用同一模板也超过监督基线和 ChatGPT 方案;还提出几种变体,可降到线性复杂度。
HN 上的人在讨论用 LLM 做成对比较排序、排序时的位置偏差和对每对评估两次等做法时引用它,认为成对比较是较准确的 LLM 排序方式。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“See the listwise approach at "Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting", Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting”
“The paper [1] I mention in the repo touches upon such methods briefly.”
“That `sort` is based on pairwise comparisons, which is one of the best ways you can do it in terms of accuracy (Qin et al., 2023: Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting).”
- Large Language Models Are Effective Text Rankers with Pairwise Ranking Prompting ↗tmoertel · 2023-07-08 · 2 分