按年被提到
20212021 · 2 次2026
机器学习与深度学习›机器学习实践与反思·arXiv:2107.07002 · cs.LG

The Benchmark Lottery

Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, Oriol Vinyals · 2021
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2021-07-17 · 最近 2021-12-06
2不同的人
2活跃月份
2讨论串
81被引(OpenAlex)
arXiv 摘要(候选层不写导读)

The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods. This paper proposes the notion of "a benchmark lottery" that describes the overall fragility of the ML benchmarking process. The benchmark lottery postulates that many factors, other than fundamental algorithmic superiority, may lead to a method being perceived as superior. On multiple benchmark setups that are prevalent in the ML community, we show that the relative performance of algorithms may be altered significantly simply by choosing different benchmark tasks, highlighting the fragility of the current paradigms and potential fallacious interpretation derived from benchmarking ML methods. Given that every benchmark makes a statement about what it perceives to be important, we argue that this might lead to biased progress in the community. We discuss the implications of the observed phenomena and provide recommendations on mitigating them using multiple machine learning domains and communities as use cases, including natural language processing, computer vision, information retrieval, recommender systems, and reinforcement learning.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论carschno · 2021-12

“Too bad they don't cite the paper "The Benchmark Lottery" (M. Dehghani et al, 2021) (https://arxiv.org/abs/2107.07002) > The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods. This paper proposes the notion of "a benchmark lottery" that describes the…”

A cartel of influential datasets are dominating machine learning research原评论 ↗
被提交为 story · 1 次