“This paper [1] shows that giving character-level awareness to the model can improve the "visual spelling".”
Character-Aware Models Improve Visual Text Rendering
给文本编码器加入字符级信息,让图像生成模型能拼对文字
当前图像生成模型很难可靠地生成格式正确的文字。论文探究一个关键因素:主流文本到图像模型缺乏字符级输入特征,难以把一个词的视觉形态预测为一串字形。为量化这一影响,作者比较字符感知与字符盲的文本编码器:纯文本领域里,字符感知模型在新的拼写任务 WikiSpell 上增益很大;应用到视觉领域,训练一组图像生成模型,字符感知版本在他们的 DrawText 基准各项文本渲染任务上都优于字符盲版本,在罕见词上比竞争者高出 30 多个点的准确率,而训练样本少得多。
HN 评论者引用它来解释图像生成模型为什么拼不对字:问题多半出在文本编码器的分词上,换成字节级模型(如 ByT5)就能明显改善拼写。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“I read a recent paper[2] that found that replacing token based text encoders in text to image models with these byte level models improves the spelling abilities of the txt2img models”
“It is most likely due to the text encoder”
“it can be experimentally established by swapping out even a very large LLM like PaLM for a humble, small, weak, but not badly-tokenized ByT5 and noting instant solution of the 'problem'.”
“Yes, this is the 'miracle of spelling' (as Character-Aware Models Improve Visual Text Rendering calls it)”
- Improving Text Rendering in Image Generation Models ↗vulkd · 2022-12-22 · 1 分