“This seems to be a good introduction to the topic: Distributed Representations of Words and Phrases and their Compositionality”
Distributed Representations of Words and Phrases and their Compositionality
Skip-gram 的降采样与负采样让词向量更好更快(word2vec)
在连续 Skip-gram 模型基础上提出若干扩展:对高频词降采样,既加快训练又使词表示更规整;提出负采样(negative sampling)作为分层 softmax 的简单替代。论文还指出词向量对词序不敏感、无法表示习语(如 Canada 与 Air 不能直接组合成 Air Canada),因此提出一个简单的短语发现方法,并表明可以为数百万个短语学到好的向量。
评论者把它当作 word2vec 的入门文本:用来解释词向量加减运算为何有效(图 2 的 PCA 投影)、负采样比分层 softmax 更高效,以及每个词有两套向量这一细节。
学习路径
读这篇之前,按顺序读书架上的 1 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出先读 word2vec 的第一篇,掌握 Skip-gram 与词向量的基本设定,再读第二篇的降采样与负采样等改进。
- 1
目标论文是它的后续,Skip-gram 模型和训练设定都来自这里。
读CBOW 与 Skip-gram 两种模型结构,以及词类比任务的评测方式。
- 2Distributed Representations of Words and Phrases and their Compositionality
目标论文:用降采样与负采样改进 Skip-gram。
读层次 softmax、负采样、高频词降采样,以及短语的识别与加法组合性。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“It relies on utilizing word embeddings (such as those described here: Distributed Representations of Words and Phrases and their Compositionality)”
“work that came out in 2013.”
“The 2-dimensional PCA projection in figure 2 of one of Mikolov's papers [1] gives an intuition why substraction/addition generally works.”
“despite that paper reporting that "negative sampling" is more computationally efficient and of similar quality.”
“the fact that each word is represented by two vectors was also mentioned in the original paper describing word2vec:”