“thought it's interesting as we hear a lot of people's experience, but this seems structured and well tracked set of specific scenarios”
How is ChatGPT's behavior changing over time?
对比 GPT-3.5/4 三月与六月版本,行为差异显著
作者在数学题、敏感问题、观点调查、多跳知识问答、代码生成、美国医师执照考试和视觉推理七类任务上,对比 GPT-3.5 与 GPT-4 的 2023 年 3 月和 6 月版本,发现同一服务的表现和行为可在短时间内大幅变化。例如 GPT-4 判断质数与合数的准确率从 84% 降到 51%,GPT-4 遵循用户指令的能力下降,是许多行为漂移背后的共同因素,因此需要对 LLM 持续监控。
评论者引用它作为 ChatGPT 在版本更新后行为漂移、可能退化的结构化证据,同时也有不少人批评任务设计不合理、只按能否执行来评判代码,夸大了退化结论。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“as if GPT-4 code generation is getting worse”
“This article is rooted in the junk science by James Zou et al, published in their paper”
“the tasks they evaluate aren't useful”
“The study was executed very poorly and should be ignored with extreme prejudice.”
“It suggests no such thing, and the problems with citation 3 have been extensively discussed.”
看全部 15 条 →
“their CTO co-authored a borderline dishonest study which got a ton of traction last summer trying to discredit GPT-4”
“From this recent study: How is ChatGPT's behavior changing over time?”
“Although performance has varied over time How is ChatGPT's behavior changing over time?”
“[Behavioral changes in ChatGPT (arXiv)]”
“which shows ChatGPT can degrade across releases, and I believe that”
“Has there been any consensus on this phenomenon”
“my source is that people have been noticing this since GPT4 days.”
“I could only really find one research paper on the topic [1], but it seems less focused on this specific issue.”
“they alter the models whenever they feel like it.”
- (Databricks cofounder) How Is ChatGPT’s Behavior Changing over Time? ↗tim_sw · 2023-07-19 · 1 分
- How is ChatGPT's behavior changing over time? ↗tim_sw · 2023-07-19 · 2 分
- How is ChatGPT's behavior changing over time? ↗adatta02 · 2023-07-20 · 1 分