按年被提到
20222022 · 1 次2026
大模型与语言模型›词向量与预训练模型·arXiv:2201.10005 · cs.CL

Text and Code Embeddings by Contrastive Pre-Training

Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam 等 25 人 · 2022
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2022-08-19 · 最近 2023-05-03
2不同的人
2活跃月份
2讨论串
381被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Text embeddings are useful features in many applications such as semantic search and computing text similarity. Previous work typically trains models customized for different use cases, varying in dataset choice, training objective and model architecture. In this work, we show that contrastive pre-training on unsupervised data at scale leads to high quality vector representations of text and code. The same unsupervised text embeddings that achieve new state-of-the-art results in linear-probe classification also display impressive semantic search capabilities and sometimes even perform competitively with fine-tuned models. On linear-probe classification accuracy averaging over 7 tasks, our best unsupervised model achieves a relative improvement of 4% and 1.8% over previous best unsupervised and supervised text embedding models respectively. The same text embeddings when evaluated on large-scale semantic search attains a relative improvement of 23.4%, 14.7%, and 10.6% over previous best unsupervised methods on MSMARCO, Natural Questions and TriviaQA benchmarks, respectively. Similarly to text embeddings, we train code embedding models on (text, code) pairs, obtaining a 20.8% relative improvement over prior best work on code search.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论roknovosel · 2022-08

“Great read, thanks for sharing. Would love to see the natural language + code mixed in there:) I've been interested in contrastive learning for a while, mainly as a means to train semantic code search models. OpenAI released a great paper on this topic called Text and Code Embeddings by Contrastive Pre-Training[1] that outlines the approach. I've used i…”

Contrastive Representation Learning原评论 ↗
评论heliophobicdude · 2023-05

“Hi Jason! I have a few thoughts on this! Refined training is usually updating the weights of usually what's called a foundational model with well structured and numerous data. It's very expensive and can disrupt the usefulness of having all the generalizations baked in from training data [1]. While LLMs can generate text based on a wide range of inputs,…”

OpenLLaMA: An Open Reproduction of LLaMA原评论 ↗