“See The False Promise of Imitating Proprietary LLMs for why calling this a private ChatGPT is probably misleading.”
The False Promise of Imitating Proprietary LLMs
指出微调模仿 ChatGPT 的开源模型只学到风格而非能力
论文批判性地分析用更强模型(如 ChatGPT)的输出去微调弱模型的做法(如 Alpaca、Self-Instruct)。作者用不同基座规模(1.5B 到 13B)、数据来源和模仿数据量(0.3M 到 150M token)微调了一系列模仿 ChatGPT 的模型,并用众包评分和标准 NLP 基准评估。起初众包工人认为输出与 ChatGPT 相当,但更有针对性的自动评估显示,在模仿数据支持不多的任务上,模仿模型几乎没有缩小基座模型与 ChatGPT 的差距,因为它们擅长模仿风格而非事实性。作者结论是模仿是虚假的承诺,最有杠杆的做法是开发更好的基座模型。
HN 上的人用它提醒:小型开源模型靠模仿 ChatGPT 看起来更好,可能只是评估被人工评分蒙蔽,风格差距很快缩小但能力差距仍然很大。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“For another possibility, see The False Promise of Imitating Proprietary LLMs. The new models may not actually be better - the evaluation may be broken.”
“Recent papers suggest that these smaller LMs dont fully capture ChatGPT quality in ways that may not have appeared with crowd worker ratings [1].”
“While the small open models have a tendency to overclaim[0], they'll get to GPT-4 level in time.”
“found imitation closes the style gap fast but there is a large capability gap.”
- The False Promise of Imitating Proprietary LLMs ↗lebek · 2023-05-26 · 18 分
- The False Promise of Imitating Proprietary LLMs ↗handfuloflight · 2026-06-25 · 1 分