mlsanityData to 7 Oct 2026

ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition

arXiv 2503.10673 ↗First posted 10 Mar 2025cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

We introduce ZeroSumEval, a dynamic, competition-based, and evolving evaluation framework for Large Language Models (LLMs) that leverages competitive games. ZeroSumEval encompasses a diverse suite of games, including security challenges (Capture the Flag), classic board games (chess), and knowledge tests (MathQuiz). These games are designed to evaluate a range of capabilities such as strategic reasoning, planning, knowledge application, safety, and adaptability. Building upon recent studies that highlight the effectiveness of game-based evaluations for LLMs, ZeroSumEval enhances these approaches by providing a standardized and extensible framework for easily implementing games and leverages DSPy to provide a better abstraction for LLM player strategies.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 20 of its 28 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

20 of 28 references.

References · 20

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  3. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  4. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  5. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  6. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  7. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  8. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  9. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
  10. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  11. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118
  12. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 118
8 more
  1. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agentscited by 89
  2. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  3. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 78
  4. 2023Elo Uncovered: Robustness and Best Practices in Language Model Evaluationcited by 58
  5. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 38
  6. 2023DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelinescited by 27
  7. 2024Evaluating Large Language Models with Grid-Based Game Competitions: An Extensible LLM Benchmark and Leaderboardcited by 22
  8. 2025Are Large Vision Language Models Good Game Players?cited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Measuring General Intelligence with Generated Gamescited by 21
  2. 2025Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratingscited by 5
  3. 2026Interactive Critique-Revision Training for Reliable Structured LLM Generationcited by 2
  4. 2025SKATE, a Scalable Tournament Eval: Weaker LLMs differentiate between stronger ones using verifiable challengescited by 1
  5. 2026Position: AI Is Not Ready for Strategic Conflictscited by 0