“If anyone's interested in this topic, here are some more foundational papers to take a look at:”
Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution
提出得分熵,让扩散模型在离散文本上追平自回归模型
扩散模型在语言等离散数据上一直表现不佳。论文提出得分熵(score entropy)这一新损失,把得分匹配自然推广到离散空间,并据此构建 SEDD 离散扩散模型。在语言建模上,SEDD 比现有语言扩散方法的困惑度降低 25%–75%,与自回归模型相当并超过 GPT-2;还可用约 32 倍更少的网络评估获得相近质量,并支持可控的填空式生成。
它被评论者当作离散(文本)扩散的奠基性工作,常出现在讨论 Inception 的 Mercury 等扩散式语言模型时,也有人为它做了独立复现。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“In the meantime, you can take a look at some of the research papers that inspired our work:”
“The authors tested their proposed method on small autoregressive language modeling tasks, and claimed promising results:”
“See equations 18-20 in Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution”
“1. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution - Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution”
“However, the terminal-masking system is clearly superior in their benchmarks.”
看全部 7 条 →
“I wrote the first (as far as I can tell) independent from-scratch reimplementation of SEDD:”