“https://arxiv.org/abs/1806.02643 and https://arxiv.org/abs/1803.06376 (as does the choice to drop the checkpoints & historical self-plays in general from the training) indicate that AlphaGo versions, at least up until then, tend to be transitive: > What is worthwhile to observe from the AlphaGo dataset, and illustrated as a series in Figures 3 and 4, is …”
Re-evaluating Evaluation
Progress in machine learning is measured by careful evaluation on problems of outstanding common interest. However, the proliferation of benchmark suites and environments, adversarial attacks, and other complications has diluted the basic evaluation model by overwhelming researchers with choices. Deliberate or accidental cherry picking is increasingly likely, and designing well-balanced evaluation suites requires increasing effort. In this paper we take a step back and propose Nash averaging. The approach builds on a detailed analysis of the algebraic structure of evaluation in two basic scenarios: agent-vs-agent and agent-vs-task. The key strength of Nash averaging is that it automatically adapts to redundancies in evaluation data, so that results are not biased by the incorporation of easy tasks or weak agents. Nash averaging thus encourages maximally inclusive evaluation -- since there is no harm (computational cost aside) from including all available tasks and agents.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“If you're interested in evaluating and rating/ranking agents, it might be worthwhile checking out DeepMind's multidimensional Elo rating system (https://arxiv.org/abs/1806.02643) which attempts to solve some of the issues with Elo and Glicko. Most notably, the ability to handle non-transitive interactions (like rock, paper, scissors) and the presence of r…”