mlsanityData to 14 Oct 2026

MindGames Arena Generalization Track: In2AI Solution with Delayed Per-Step Reward Attribution

arXiv 2606.00017 ↗First posted 13 Apr 2026cs.AI cs.CL cs.MAcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Training language model agents for multi-agent strategic interaction presents a core difficulty: the quality of any action may depend on future events that never materialize, on moves that violate game rules, or on decisions made by other players. Standard reinforcement learning assumes that rewards can be assigned at each step, but this assumption fails in settings where outcomes are entangled across time and agents. We introduce delayed per-step reward attribution with eligibility gating, an episode lifecycle and postprocessing pipeline that computes rewards only at episode end, propagates them back to originating steps according to task-specific semantics, and excludes steps that lack valid dependent information from training. Together with asynchronous rollout generation via vLLM's continuous batching, curriculum-based opponent sampling, and multi-level stratified batch construction, this approach enables stable, sample-efficient RL training in multi-agent environments. We evaluate on the MindGames Arena benchmark at NeurIPS 2025, where a single 8-billion-parameter open-source model trained with our method matched or surpassed substantially larger proprietary systems, including GPT-5, in head-to-head play and took first place in both the Open (unrestricted) and Efficient (<=8B parameters) tracks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 10 of its 23 references that are in the map. First posted on arXiv 13 Apr 2026.

Builds on

10 of 23 references.

References · 10

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  5. 2015Prioritized Experience Replaycited by 1,099
  6. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  7. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  8. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 264
  9. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 150
  10. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?cited by 24

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ArenaFlow: From Trajectory Ranking to Hierarchical Credit Propagation for Open-Ended Agent RL
  2. 2025Self-correcting Reward Shaping via Language Models for Reinforcement Learning Agents in Games
  3. 20252048: Reinforcement Learning in a Delayed Reward Environment
  4. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution
  5. 2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents
  6. 2025Reinforced Language Models for Sequential Decision Making
  7. 2024Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback
  8. 2026RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Models