“Differentiale search indices go into this direction: https://arxiv.org/abs/2202.06991 The approach in the paper has rough edges, but the metrics are bonkers (double digit percentage POINTS improvement over dual encoders). This paper was written before the LLM craze, and I am not aware of any further developments in that area. I think that this area might…”
Transformer Memory as a Differentiable Search Index
In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model. To this end, we introduce the Differentiable Search Index (DSI), a new paradigm that learns a text-to-text model that maps string queries directly to relevant docids; in other words, a DSI model answers queries directly using only its parameters, dramatically simplifying the whole retrieval process. We study variations in how documents and their identifiers are represented, variations in training procedures, and the interplay between models and corpus sizes. Experiments demonstrate that given appropriate design choices, DSI significantly outperforms strong baselines such as dual encoder models. Moreover, DSI demonstrates strong generalization capabilities, outperforming a BM25 baseline in a zero-shot setup.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Another potent alternative is perhaps Differentiable Search Index (DSI) based on Transformer Memory: Transformer Memory as a Differentiable Search Index: https://arxiv.org/abs/2202.06991”
- Transformer Memory as a Differentiable Search Index ↗georgehill · 2022-02-16 · 80 分