引用moyix · 2022-07
“See e.g. the NeurIPS 2014 and 2021 consistency experiments: Inconsistency in Conference Peer Review, [LINK]”
引用 NeurIPS 2014 和 2021 一致性实验,说明同行评审不一致
重访 NeurIPS 2014 评审实验:评审擅长识别差论文而非好论文
论文重访了 2014 年考察会议同行评审不一致性的 NeurIPS 实验,确定评审质量分数的变异中有 50% 源自主观因素。七年后发现,对被接收的论文,质量分数与以引用量衡量的影响力没有相关性;追踪被拒论文最终发表去向后则发现二者存在相关。作者据此认为当年的评审善于发现差论文、却不善于发现好论文,并提醒不要因此去掉主观因素,还建议评价研究者个人时少看重“顶会论文”。
HN 上的人引用它作为同行评审不稳定的证据,尤其是“评审擅长识别差论文却不擅长识别好论文”这一结论。
“See e.g. the NeurIPS 2014 and 2021 consistency experiments: Inconsistency in Conference Peer Review, [LINK]”
“But the Google paper[1] notes that reviewers are "good at identifying bad papers but not good at identifying good papers."”
“[1] Inconsistency in Conference Peer Review”
“[3.2] A follow-up on the 2014 experiment tdlr: reviewers are good at identifying bad papers, but not good at identifying good papers (i.e. bias to reject): Inconsistency in Conference Peer Review”