引用nojvek · 2023-10
“Related paper Pretraining on the Test Set Is All You Need”
贴出相关论文,说明在测试集上预训练会抬高基准分数
讽刺论文:在测试集上预训练,百万参数模型在基准上拿满分
这是一篇讽刺论文:作者专门用评测基准构造不到 10 万 token 的“高质量非合成”数据混合,预训练出只有 100 万参数的 phi-CTNL,声称它在多个学术基准上取得满分、严格优于所有已知基础模型,还打破幂律缩放并表现出能准确预测基准“金丝雀”字符串的类 grokking 能力。借此嘲讽基准污染和刷榜。
评论者把它当作测试集污染和基准刷分的梗来引用,包括用它调侃 Grok 3 的榜单成绩。
“Related paper Pretraining on the Test Set Is All You Need”
“it's just contaminated with this specific prompt.”
“Pre-training on the test set is all you need. Pretraining on the Test Set Is All You Need”
“Time to review Pretraining on the Test Set Is All You Need AI-CEO.org's best friend”