mlsanityData to 14 Oct 2026

Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot

arXiv 2107.06857 ↗First posted 14 Jul 2021cs.MA cs.AIcited by 99

Not on the map. Why?

Cited by 24 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing evaluation suites for multi-agent reinforcement learning (MARL) do not assess generalization to novel situations as their primary objective (unlike supervised-learning benchmarks). Our contribution, Melting Pot, is a MARL evaluation suite that fills this gap, and uses reinforcement learning to reduce the human labor required to create novel test scenarios. This works because one agent's behavior constitutes (part of) another agent's environment. To demonstrate scalability, we have created over 80 unique test scenarios covering a broad range of research topics such as social dilemmas, reciprocity, resource sharing, and task partitioning. We apply these test scenarios to standard MARL training algorithms, and demonstrate how Melting Pot reveals weaknesses not apparent from training performance alone.

Key facts. Cited by 99 papers in the map (data through 14 Oct 2026). Builds on 56 of its 112 references that are in the map. First posted on arXiv 14 Jul 2021.

Builds on

56 of 112 references.

References · 56

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  3. 2016OpenAI Gymcited by 1,736
  4. 2016Concrete Problems in AI Safetycited by 1,642
  5. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  7. 2018DeepMind Control Suitecited by 956
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  9. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
  10. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  11. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  12. 2018Quantifying Generalization in Reinforcement Learningcited by 352
38 more
  1. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 349
  2. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  3. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 313
  4. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  5. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 268
  6. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 264
  7. 2017Emergence of Grounded Compositional Language in Multi-Agent Populationscited by 263
  8. 2017Learning with Opponent-Learning Awarenesscited by 222
  9. 2020"Other-Play" for Zero-Shot Coordinationcited by 198
  10. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 192
  11. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 186
  12. 2020Emergent Multi-Agent Communication in the Deep Learning Eracited by 170
  13. 2020Open Problems in Cooperative AIcited by 168
  14. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 141
  15. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 141
  16. 2019Behaviour Suite for Reinforcement Learningcited by 132
  17. 2017Emergent Complexity via Multi-Agent Competitioncited by 120
  18. 2018Generalization and Regularization in DQNcited by 118
  19. 2018Inequity aversion improves cooperation in intertemporal social dilemmascited by 106
  20. 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
  21. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 87
  22. 2019AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligencecited by 86
  23. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  24. 2020Towards Playing Full MOBA Games with Deep Reinforcement Learningcited by 78
  25. 2016Research Priorities for Robust and Beneficial Artificial Intelligencecited by 74
  26. 2019On the Pitfalls of Measuring Emergent Communicationcited by 71
  27. 2017A multi-agent reinforcement learning model of common-pool resource appropriationcited by 69
  28. 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 67
  29. 2019Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Researchcited by 61
  30. 2019Emergent Coordination Through Competitioncited by 57
  31. 2019Obstacle Tower: A Generalization Challenge in Vision, Control, and Planningcited by 54
  32. 2020Social diversity and social preferences in mixed-motive reinforcement learningcited by 49
  33. 2017Maintaining cooperation in complex social dilemmas using deep reinforcement learningcited by 48
  34. 2017Prosocial learning agents solve generalized Stag Hunts better than selfish onescited by 47
  35. 2018Re-evaluating Evaluationcited by 47
  36. 2019Generalization of Reinforcement Learners with Working and Episodic Memorycited by 36
  37. 2018Natural Environment Benchmarks for Reinforcement Learningcited by 32
  38. 2020Too many cooks: Bayesian inference for coordinating multi-agent collaborationcited by 29

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 13; 2023: 23; 2024: 20; 2025: 18; 2026: 25.

99 papers.

Sort:

By area:

  1. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  2. 2023Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordiacited by 126
  3. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 122
  4. 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 63
  5. 2022Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real worldcited by 56
  6. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 49
  7. 2023Welfare Diplomacy: Benchmarking Language Model Cooperationcited by 45
  8. 2022A Survey of Ad Hoc Teamwork Researchcited by 43
  9. 2025Infrastructure for AI Agentscited by 41
  10. 2022Melting Pot 2.0cited by 40
  11. 2024Moral Alignment for LLM Agentscited by 39
  12. 2023BenchMARL: Benchmarking Multi-Agent Reinforcement Learningcited by 33
38 more
  1. 2023A Review of Cooperation in Multi-agent Learningcited by 29
  2. 2025A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectivescited by 27
  3. 2022MARLlib: A Scalable and Efficient Multi-agent Reinforcement Learning Librarycited by 21
  4. 2024TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraftcited by 21
  5. 2024A social path to human-like artificial intelligencecited by 20
  6. 2022Generalization in Cooperative Multi-Agent Systemscited by 19
  7. 2024MineLand: Simulating Large-Scale Multi-Agent Interactions with Limited Multimodal Senses and Physical Needscited by 18
  8. 2024N-Agent Ad Hoc Teamworkcited by 15
  9. 2023Games for Artificial Intelligence Research: A Review and Perspectivescited by 14
  10. 2025The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperationcited by 13
  11. 2023Towards the Scalable Evaluation of Cooperativeness in Language Modelscited by 12
  12. 2023Scaling Opponent Shaping to High Dimensional Gamescited by 12
  13. 2025Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordiacited by 12
  14. 2026AI Gamestore: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Gamescited by 12
  15. 2024Learning and Sustaining Shared Normative Systems via Bayesian Rule Induction in Markov Gamescited by 11
  16. 2024Can LLM-Augmented autonomous agents cooperate?, An evaluation of their cooperative capabilities through Melting Potcited by 11
  17. 2024Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarkscited by 11
  18. 2022Stateful active facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learningcited by 10
  19. 2023Hardness of Independent Learning and Sparse Equilibrium Computation in Markov Gamescited by 10
  20. 2023Information Design in Multi-Agent Reinforcement Learningcited by 9
  21. 2023Neural MMO 2.0: A Massively Multi-task Addition to Massively Multi-agent Learningcited by 9
  22. 2024Your Co-Workers Matter: Evaluating Collaborative Capabilities of Language Models in Blocks Worldcited by 9
  23. 2024Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learningcited by 8
  24. 2025SocialJax: An Evaluation Suite for Multi-agent Reinforcement Learning in Sequential Social Dilemmascited by 8
  25. 2026M3-BENCH: Process-Aware Evaluation of LLM Agents' Social Behaviors in Mixed-Motive Gamescited by 8
  26. 2022Hidden Agenda: a Social Deduction Game with Diverse Learned Equilibriacited by 7
  27. 2023RaidEnv: Exploring New Challenges in Automated Content Balancing for Boss Raid Gamescited by 7
  28. 2023Framework-Based Qualitative Analysis of Free Responses of Large Language Models: Algorithmic Fidelitycited by 7
  29. 2025Rethinking Agent Design: From Top-Down Workflows to Bottom-Up Skill Evolutioncited by 6
  30. 2025VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environmentscited by 6
  31. 2025Beyond Statistical Learning: Exact Learning Is Essential for General Intelligencecited by 6
  32. 2026Generative AI collective behavior needs an interactionist paradigmcited by 6
  33. 2022Dynamic Collaborative Multi-Agent Reinforcement Learning Communication for Autonomous Drone Reforestationcited by 5
  34. 2025End-to-end PDDL Planning with Hardcoded and Dynamic Agentscited by 5
  35. 2023Co-Learning Empirical Games and World Modelscited by 4
  36. 2024Neural Population Learning beyond Symmetric Zero-sum Gamescited by 4
  37. 2024Normative Modules: A Generative Agent Architecture for Learning Norms that Supports Multi-Agent Cooperationcited by 4
  38. 2024Craftium: Bridging Flexibility and Efficiency for Rich 3D Single- and Multi-Agent Environmentscited by 4

50 most cited of 99 All 99 (JSON)