按年被提到
20242025 · 3 次2026
视觉、语音与图像生成›扩散模型·arXiv:2406.07524 · cs.CL

Simple and Effective Masked Diffusion Language Models

Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, Volodymyr Kuleshov · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2025-02-21 · 最近 2025-05-01
3不同的人
2活跃月份
3讨论串
75被引(OpenAlex)
arXiv 摘要(候选层不写导读)

While diffusion models excel at generating high-quality images, prior work reports a significant performance gap between diffusion and autoregressive (AR) methods in language modeling. In this work, we show that simple masked discrete diffusion is more performant than previously thought. We apply an effective training recipe that improves the performance of masked diffusion models and derive a simplified, Rao-Blackwellized objective that results in additional improvements. Our objective has a simple form -- it is a mixture of classical masked language modeling losses -- and can be used to train encoder-only language models that admit efficient samplers, including ones that can generate arbitrary lengths of text semi-autoregressively like a traditional language model. On language modeling benchmarks, a range of masked diffusion models trained with modern engineering practices achieves a new state-of-the-art among diffusion models, and approaches AR perplexity. We provide the code, along with a blog post and video tutorial on the project page: https://s-sahoo.com/mdlm

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论jxmorris12 · 2025-02

“Thanks for bringing this up! As far as I understand it current text diffusion models are limited to fairly short context windows. The idea of a text diffusion model continuously updating and revising a million-token-long chain-of-thought is pretty mind-boggling. I agree that these non-autoregressive models could potentially behave in completely different …”

I think Yann Lecun was right about LLMs (but perhaps only by accident)原评论 ↗
评论volodia · 2025-02

“Not today, but we will be following up with a technical report over the next week or so. In the meantime, you can take a look at some of the research papers that inspired our work: - https://arxiv.org/abs/2310.16834 - https://arxiv.org/abs/2406.07524”

Mercury Coder: frontier diffusion LLM generating 1000+ tok/sec on commodity GPUs原评论 ↗
评论freeqaz · 2025-05

“Here are two papers linked from Inception's site: 1. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution - https://arxiv.org/abs/2310.16834 2. Simple and Effective Masked Diffusion Language Models - https://arxiv.org/abs/2406.07524”

Mercury: Commercial-scale diffusion language model原评论 ↗