按年被提到
20232023 · 11 次2026
大模型与语言模型›对话式大模型·arXiv:2307.09009 · cs.CL

How is ChatGPT's behavior changing over time?

ChatGPT 的行为如何随时间变化?
Lingjiao Chen, Matei Zaharia, James Zou · 2023

对比 GPT-3.5/4 三月与六月版本,行为差异显著

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 17 人首次 2023-07-19 · 最近 2026-09-28
17不同的人
9活跃月份
18讨论串
657被引(OpenAlex)

作者在数学题、敏感问题、观点调查、多跳知识问答、代码生成、美国医师执照考试和视觉推理七类任务上,对比 GPT-3.5 与 GPT-4 的 2023 年 3 月和 6 月版本,发现同一服务的表现和行为可在短时间内大幅变化。例如 GPT-4 判断质数与合数的准确率从 84% 降到 51%,GPT-4 遵循用户指令的能力下降,是许多行为漂移背后的共同因素,因此需要对 LLM 持续监控。

HN 为什么提它

评论者引用它作为 ChatGPT 在版本更新后行为漂移、可能退化的结构化证据,同时也有不少人批评任务设计不合理、只按能否执行来评判代码,夸大了退化结论。

实证研究难度 2 / 5约 25 分钟有时效性
读之前最好知道
1GPT-3.5 与 GPT-4 / ChatGPT 的基本使用
2基准评测方法
3思维链提示(chain-of-thought prompting)

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐NikolaNovak · 2023-09

“thought it's interesting as we hear a lot of people's experience, but this seems structured and well tracked set of specific scenarios”

认为这项研究有意思,用结构化、有跟踪的具体场景,而不只是个人体验
Study shows over time, ChatGPT lost accuracy原评论 ↗
批评lhl · 2023-07

“as if GPT-4 code generation is getting worse”

批评它只按代码能否直接运行来判断对错,夸大了 GPT-4 代码生成变差的说法
GPT-4 is getting worse over time, not better原评论 ↗
批评ndr_ · 2023-08

“This article is rooted in the junk science by James Zou et al, published in their paper”

Why ChatGPT Is Getting Dumber at Basic Math原评论 ↗
批评n2d4 · 2023-08

“the tasks they evaluate aren't useful”

After 118 days, GPT Unicorn has produced a unicorn原评论 ↗
批评modeless · 2023-08

“The study was executed very poorly and should be ignored with extreme prejudice.”

Show HN: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B原评论 ↗
批评gwern · 2024-02

“It suggests no such thing, and the problems with citation 3 have been extensively discussed.”

Generative AI Degrades Online Communities原评论 ↗
看全部 15 条 →
批评BoorishBears · 2024-03

“their CTO co-authored a borderline dishonest study which got a ton of traction last summer trying to discredit GPT-4”

DBRX: A new open LLM原评论 ↗
引用embeng4096 · 2023-07

“From this recent study: How is ChatGPT's behavior changing over time?”

引用它:GPT-4 判断质数的准确率从 97.6% 降到 2.4%
Ask HN: Is SICP/HtDP still worth reading in 2023? Any alternatives?原评论 ↗
引用shubb · 2023-08

“Although performance has varied over time How is ChatGPT's behavior changing over time?”

引作证据:ChatGPT 的表现会随模型版本变化
Azure ChatGPT: Private and secure ChatGPT for internal enterprise use原评论 ↗
引用ocolegro · 2023-08

“[Behavioral changes in ChatGPT (arXiv)]”

贴出研究 ChatGPT 行为在不同版本间变化的论文
The AI Reproducibility Crisis in GPT-3.5/GPT-4 Research原评论 ↗
引用armchairhacker · 2024-08

“which shows ChatGPT can degrade across releases, and I believe that”

引用它:ChatGPT 在不同发布版本之间可能退化
ChatGPT quality strangely gotten worse原评论 ↗
引用omega3 · 2024-08

“Has there been any consensus on this phenomenon”

把它列在关于 ChatGPT 退化的链接中,并询问这个现象是否已有共识
Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?原评论 ↗
引用b65e8bee43c2ed0 · 2026-04

“my source is that people have been noticing this since GPT4 days.”

引作证据:ChatGPT 的行为确实随时间漂移
Claude Opus 4.7 costs 20–30% more per session原评论 ↗
引用sporkland · 2026-06

“I could only really find one research paper on the topic [1], but it seems less focused on this specific issue.”

说只找到这一篇相关研究,但似乎不太针对这个具体问题
Ask HN: Agents get dumber before release of new model version?原评论 ↗
引用vlyan · 2026-09

“they alter the models whenever they feel like it.”

引用它说明厂商想改就改模型,ChatGPT 的行为随更新变化
被提交为 story · 3 次