“our recent work[0] actually found it may cause some performance degradation as well as possible reasoning weakening in JSON”
Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models
发现强制 JSON 等结构化输出格式会明显削弱 LLM 的推理能力
论文研究 JSON、XML 等标准化格式的结构化生成是否会影响 LLM 的推理和领域知识理解能力。作者在多种常见任务上对比模型在被限制遵守结构化格式与自由生成时的表现,意外地观察到在格式限制下 LLM 的推理能力显著下降;并且格式约束越严格,推理任务上的性能下降通常越大。
HN 上的人在讨论强制 JSON 输出或受限解码时引用它,作为格式限制可能降低推理质量的证据,同时也有人提醒结果与具体模型等因素有关。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Research Paper, is on maintaining the model’s accuracy while producing structured outputs.”
“Furthermore, we find that stricter format constraints generally lead to greater performance degradation in reasoning tasks.”
“I'm not sure how true this is, but when using "forced" json output it def had a big drop off in quality - Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models.”
“There is work that suggests that constrained decoding can negatively affect model quality, but this depends on many factors, among which the model in question [1, 2, 3].”
- A Study on the Impact of Structured Output on Performance of LLMs ↗blackcat201 · 2024-08-07 · 2 分
- A Study on the Impact of Format Restrictions on Performance of LLMs ↗kmdupree · 2024-08-07 · 1 分
- Study on the Impact of Format Restrictions on Performance of LMM ↗danfritz · 2024-08-13 · 1 分
- A Study on the Impact of Format Restrictions on Performance of LLM ↗quxinxin · 2024-08-16 · 1 分
- A Study on the Impact of Format Restrictions on Performance of LLM ↗handfuloflight · 2024-08-31 · 2 分