引用impossiblefork · 2023-03
“I got linked this paper ('Locally typical sampling' Locally Typical Sampling) when I asked some people why beam search only works for the kind of stuff above.”
有人给了这篇论文来回答为什么 beam search 只适用于某类任务
用信息论视角提出局部典型采样,减少文本生成中的重复
论文把自然语言生成抽象为离散随机过程,用信息论分析解释为何高概率文本可能乏味或重复。作者形式化定义了每个词的信息量接近模型条件熵的字符串集合,并据此提出简单高效的局部典型采样(locally typical sampling)。自动与人工评估显示,在摘要与故事生成上,其质量与 nucleus、top-k 采样相当,并能稳定地减少退化的重复。
评论者用它解释为何 beam search 对语言模型效果不佳、作为替代解码方法,并说明温度为 0 的高概率文本缺乏“意外感”、显得不自然。
“I got linked this paper ('Locally typical sampling' Locally Typical Sampling) when I asked some people why beam search only works for the kind of stuff above.”
“There are other decoding approaches, other than beam search:”
“are doing far FAR more to advance the state of AI than dozens of VC backed LLM companies operating today.”
“Temperature 0 generated text actually has a weird "lack of surprise" character that makes it seem artificial. [1]”