mlsanityData to 14 Oct 2026

Breaking the Performance Ceiling in Reinforcement Learning requires Inference Strategies

arXiv 2505.21236 ↗First posted 27 May 2025cs.LG cs.AI cs.MAcited by 2

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Reinforcement learning (RL) systems have countless applications, from energy-grid management to protein design. However, such real-world scenarios are often extremely difficult, combinatorial in nature, and require complex coordination between multiple agents. This level of complexity can cause even state-of-the-art RL systems, trained until convergence, to hit a performance ceiling which they are unable to break out of with zero-shot inference. Meanwhile, many digital or simulation-based applications allow for an inference phase that utilises a specific time and compute budget to explore multiple attempts before outputting a final solution. In this work, we show that such an inference phase employed at execution time, and the choice of a corresponding inference strategy, are key to breaking the performance ceiling observed in complex multi-agent RL problems. Our main result is striking: we can obtain up to a 126% and, on average, a 45% improvement over the previous state-of-the-art across 17 tasks, using only a couple seconds of extra wall-clock time during execution. We also demonstrate promising compute scaling properties, supported by over 60k experiments, making it the largest study on inference strategies for complex RL to date. Our experimental data and code are available at https://sites.google.com/view/inference-strategies-rl.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 44 of its 67 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

44 of 67 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning26 Feb

Other references · 43

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2025s1: Simple test-time scalingcited by 1,341
  6. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  8. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  9. 2016Neural Combinatorial Optimization with Reinforcement Learningcited by 608
  10. 2019The StarCraft Multi-Agent Challengecited by 597
  11. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  12. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
31 more
  1. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 294
  2. 2017Molecular De Novo Design through Deep Reinforcement Learningcited by 267
  3. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  4. 2013The Complexity of Decentralized Control of Markov Decision Processescited by 231
  5. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 212
  6. 2020"Other-Play" for Zero-Shot Coordinationcited by 198
  7. 2017Deep Reinforcement Learning for De-Novo Drug Designcited by 161
  8. 2022DIMES: A Differentiable Meta Solver for Combinatorial Optimization Problemscited by 129
  9. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 122
  10. 2023Hypothesis Search: Inductive Reasoning with Language Modelscited by 120
  11. 2017Quality and Diversity Optimization: A Unifying Modular Frameworkcited by 113
  12. 2021Efficient Active Search for Combinatorial Optimization Problemscited by 100
  13. 2020RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Realcited by 91
  14. 2022Simulation-guided Beam Search for Neural Combinatorial Optimizationcited by 84
  15. 2020One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RLcited by 79
  16. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 79
  17. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 77
  18. 2022Winner Takes It All: Training Performant RL Populations for Combinatorial Optimizationcited by 58
  19. 2022A Survey of Ad Hoc Teamwork Researchcited by 43
  20. 2023Combinatorial Optimization with Policy Adaptation using Latent Space Searchcited by 41
  21. 2024PolyNet: Learning Diverse Solution Strategies for Neural Combinatorial Optimizationcited by 35
  22. 2018Ranked Reward: Enabling Self-Play Reinforcement Learning for Combinatorial Optimizationcited by 27
  23. 2022Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discoverycited by 23
  24. 2023QDax: A Library for Quality-Diversity and Population-based Algorithms with Hardware Accelerationcited by 17
  25. 2021Mava: a research library for distributed multi-agent reinforcement learning in JAXcited by 16
  26. 2024N-Agent Ad Hoc Teamworkcited by 15
  27. 2024Sable: a Performant, Efficient and Scalable Sequence Model for MARLcited by 10
  28. 2024SPO: Sequential Monte Carlo Policy Optimisationcited by 5
  29. 2024Memory-Enhanced Neural Solvers for Routing Problemscited by 4
  30. 2024Role Play: Learning Adaptive Role-Specific Strategies in Multi-Agent Interactionscited by 3
  31. 2024Symmetry-Breaking Augmentations for Ad Hoc Teamworkcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Opportunities and Challenges of Natural Language Processing for Low-Resource Senegalese Languages in Social Science Researchcited by 0
  2. 2026CoScale-RL: Efficient Post-Training by Co-Scaling Data and Computationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Does Scaling Reinforcement Learning Really Require More Training?
  2. 2026Hybrid-Adaptive Thread Tuning to Mitigate Simulation Execution Bottlenecks in High-Performance Reinforcement Learning Inference
  3. 2024Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference
  4. 2025Gym4ReaL: A Suite for Benchmarking Real-World Reinforcement Learning
  5. 2026Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling
  6. 2026TMAS: Scaling Test-Time Compute via Multi-Agent Synergy
  7. 2025Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signals
  8. 2025BroRL: Scaling Reinforcement Learning via Broadened Exploration