mlsanityData to 7 Oct 2026

Re-evaluating Evaluation

arXiv 1806.02643 ↗First posted 7 Jun 2018cs.LG cs.GT stat.MLcited by 47

Not on the map. Why?

Abstract (arXiv)

Progress in machine learning is measured by careful evaluation on problems of outstanding common interest. However, the proliferation of benchmark suites and environments, adversarial attacks, and other complications has diluted the basic evaluation model by overwhelming researchers with choices. Deliberate or accidental cherry picking is increasingly likely, and designing well-balanced evaluation suites requires increasing effort. In this paper we take a step back and propose Nash averaging. The approach builds on a detailed analysis of the algebraic structure of evaluation in two basic scenarios: agent-vs-agent and agent-vs-task. The key strength of Nash averaging is that it automatically adapts to redundancies in evaluation data, so that results are not biased by the incorporation of easy tasks or weak agents. Nash averaging thus encourages maximally inclusive evaluation -- since there is no harm (computational cost aside) from including all available tasks and agents.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 5; 2023: 6; 2024: 6; 2025: 11; 2026: 6.

47 papers.

Sort:

By area:

  1. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  2. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  3. 2021The Benchmark Lotterycited by 133
  4. 2021Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Potcited by 97
  5. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  6. 2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
  7. 2021Scaling Scaling Laws with Board Gamescited by 62
  8. 2021Modelling Behavioural Diversity for Learning in Open-Ended Gamescited by 39
  9. 2021Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systemscited by 24
  10. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignmentcited by 24
  11. 2022On the Limitations of Elo: Real-World Games, are Transitive, not Additivecited by 23
  12. 2021Pick Your Battles: Interaction Graphs as Population-Level Objectives for Strategic Diversitycited by 22
35 more
  1. 2021Multi-Agent Training beyond Zero-Sum with Correlated Equilibrium Meta-Solverscited by 21
  2. 2024Position: Benchmarking is Limited in Reinforcement Learning Researchcited by 15
  3. 2025Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordiacited by 11
  4. 2021Measuring the Non-Transitivity in Chesscited by 10
  5. 2024Policy Space Response Oracles: A Surveycited by 10
  6. 2025Re-evaluating Open-ended Evaluation of Large Language Modelscited by 8
  7. 2023Combining Tree-Search, Generative Models, and Nash Bargaining Concepts in Game-Theoretic Reinforcement Learningcited by 6
  8. 2025Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoptioncited by 6
  9. 2024Position: Embracing Negative Results in Machine Learningcited by 5
  10. 2025Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratingscited by 5
  11. 2021Estimating α-Rank by Maximizing Information Gaincited by 4
  12. 2023Co-Learning Empirical Games and World Modelscited by 4
  13. 2022Game Theoretic Rating in N-player general-sum games with Equilibriacited by 3
  14. 2024Soft Condorcet Optimization for Ranking of General Agentscited by 3
  15. 2022Learning to Identify Top Elo Ratings: A Dueling Bandits Approachcited by 2
  16. 2022Developing, Evaluating and Scaling Learning Agents in Multi-Agent Environmentscited by 2
  17. 2023Ordinal Potential-based Player Ratingcited by 2
  18. 2025Is Elo Rating Reliable? A Study Under Model Misspecificationcited by 2
  19. 2026Which Nash Equilibrium? Solver-Dependent Selection on Zero-Sum Nash Polytopescited by 2
  20. 2021Learning to Play Soccer From Scratch: Sample-Efficient Emergent Coordination through Curriculum-Learning and Competitioncited by 1
  21. 2021AI in Human-computer Gaming: Techniques, Challenges and Opportunitiescited by 1
  22. 2023Differentiable Arbitrating in Zero-sum Markov Gamescited by 1
  23. 2023Composing Efficient, Robust Tests for Policy Selectioncited by 1
  24. 2025Clone-Robust Weights in Metric Spaces: Handling Redundancy Bias for Benchmark Aggregationcited by 1
  25. 2025Strength Estimation and Human-Like Strength Adjustment in Gamescited by 1
  26. 2025Adaptive Preference Aggregationcited by 1
  27. 2026Active Evaluation of General Agents: Problem Definition and Comparison of Baseline Algorithmscited by 1
  28. 2026Revisiting Regularized Policy Optimization for Stable and Efficient Reinforcement Learning in Two-Player Gamescited by 1
  29. 2022Principal Trade-off Analysiscited by 0
  30. 2024Identifying and Clustering Counter Relationships of Team Compositions in PvP Games for Efficient Balance Analysiscited by 0
  31. 2025Ranking Joint Policies in Dynamic Games using Evolutionary Dynamicscited by 0
  32. 2025Beyond the high score: Prosocial ability profiles of multi-agent populationscited by 0
  33. 2026Scorio.jl: A Julia package for ranking stochastic responsescited by 0
  34. 2026Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignmentcited by 0
  35. 2026Gradland: On Phenomenal Experience, Differentiated Across Many Dimensionscited by 0