按年被提到
20222024 · 2 次2026
大模型与语言模型›词向量与预训练模型·arXiv:2202.06991 · cs.CL

Transformer Memory as a Differentiable Search Index

Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen 等 13 人 · 2022
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2022-02-16 · 最近 2024-04-16
3不同的人
2活跃月份
3讨论串
159被引(OpenAlex)
arXiv 摘要(候选层不写导读)

In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model. To this end, we introduce the Differentiable Search Index (DSI), a new paradigm that learns a text-to-text model that maps string queries directly to relevant docids; in other words, a DSI model answers queries directly using only its parameters, dramatically simplifying the whole retrieval process. We study variations in how documents and their identifiers are represented, variations in training procedures, and the interplay between models and corpus sizes. Experiments demonstrate that given appropriate design choices, DSI significantly outperforms strong baselines such as dual encoder models. Moreover, DSI demonstrates strong generalization capabilities, outperforming a BM25 baseline in a zero-shot setup.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论haizhung · 2024-04

“Differentiale search indices go into this direction: https://arxiv.org/abs/2202.06991 The approach in the paper has rough edges, but the metrics are bonkers (double digit percentage POINTS improvement over dual encoders). This paper was written before the LLM craze, and I am not aware of any further developments in that area. I think that this area might…”

Dot – A standalone open source app meant for easy use of local LLMs and RAG原评论 ↗
评论teleforce · 2024-04

“Another potent alternative is perhaps Differentiable Search Index (DSI) based on Transformer Memory: Transformer Memory as a Differentiable Search Index: https://arxiv.org/abs/2202.06991”

Ask HN: Is RAG the Future of LLMs?原评论 ↗
被提交为 story · 1 次