按年被提到
20182018 · 1 次2026
机器学习与深度学习›机器学习实践与反思·arXiv:1806.02643 · cs.LG

Re-evaluating Evaluation

David Balduzzi, Karl Tuyls, Julien Perolat, Thore Graepel · 2018
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2018-12-06 · 最近 2020-07-21
2不同的人
2活跃月份
2讨论串
39被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Progress in machine learning is measured by careful evaluation on problems of outstanding common interest. However, the proliferation of benchmark suites and environments, adversarial attacks, and other complications has diluted the basic evaluation model by overwhelming researchers with choices. Deliberate or accidental cherry picking is increasingly likely, and designing well-balanced evaluation suites requires increasing effort. In this paper we take a step back and propose Nash averaging. The approach builds on a detailed analysis of the algebraic structure of evaluation in two basic scenarios: agent-vs-agent and agent-vs-task. The key strength of Nash averaging is that it automatically adapts to redundancies in evaluation data, so that results are not biased by the incorporation of easy tasks or weak agents. Nash averaging thus encourages maximally inclusive evaluation -- since there is no harm (computational cost aside) from including all available tasks and agents.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论gwern · 2018-12

“https://arxiv.org/abs/1806.02643 and https://arxiv.org/abs/1803.06376 (as does the choice to drop the checkpoints & historical self-plays in general from the training) indicate that AlphaGo versions, at least up until then, tend to be transitive: > What is worthwhile to observe from the AlphaGo dataset, and illustrated as a series in Figures 3 and 4, is …”

Deepmind Produces a General-Purpose Games-Playing Machine原评论 ↗
评论dcl · 2020-07

“If you're interested in evaluating and rating/ranking agents, it might be worthwhile checking out DeepMind's multidimensional Elo rating system (https://arxiv.org/abs/1806.02643) which attempts to solve some of the issues with Elo and Glicko. Most notably, the ability to handle non-transitive interactions (like rock, paper, scissors) and the presence of r…”

Elo sucks – better multiplayer rating systems for smaller games (2019)原评论 ↗