“The trend now is to make them multi language, multi modal, multi task, multi memory ([1] and this paper)”
Improving language models by retrieving from trillions of tokens
RETRO:用万亿级 token 检索库增强自回归语言模型
论文让自回归语言模型以此前 token 的局部相似度从大语料库中检索文档块,并以此为条件生成,提出 Retrieval-Enhanced Transformer(RETRO)。在 2 万亿 token 的数据库下,RETRO 在 Pile 上以少 25 倍的参数取得与 GPT-3 和 Jurassic-1 相当的表现;它结合冻结的 BERT 检索器、可微编码器和分块交叉注意力,可从头训练,也可对已有 Transformer 做 RETROfit,微调后还能用于问答等知识密集任务。
评论者把它当作“检索增强、外部记忆让小模型媲美大模型”的代表,也拿它讨论训练集与测试集的数据泄漏计算和 MassiveText 语料。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“In recent literature, the RETRO Transformer talks about "quantifying dataset leakage", which is related to what I mentioned in the above paragraph.”
“and the RETRO transformer has a calculation for the effect of "dataset leakage".”
“If you are talking about Figure 1 of Improving language models by retrieving from trillions of tokens, you are reading it wrong. Y axis is bits-per-byte and lower is better.”
“11B parameters, so I guess it's FLAN-T5[1]? combined with retrieval transformer (something like RETRO[2])?”
“or Transformers paper [2]?”
看全部 12 条 →
“Here is example of one from Google: RETRO[1][2]”
“I would be excited about an open-source RETRO model with comparable performance to GPT-3”
“Examples of such architectures:”
“DeepMind's RETRO paper Improving language models by retrieving from trillions of tokens mentions a dataset called MassiveText, which includes 20 million books of 3T tokens.”
“From what I gather the cross-attention allows the expert selection to react to the context, unlike the more naive plain Softmax gating approach.”
“This paper from DeepMind a few years ago offers a counter example to this claim.”