mlsanityData to 14 Oct 2026

Long-Horizon RL Needs Muon

arXiv 2610.20256 ↗First posted 14 Oct 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Long-horizon agents must coordinate reasoning, tool use, and environment feedback over extended interactions. Learning from sparse outcome rewards requires weighting trajectories of different lengths and updating the policy while rollouts are still being collected. We examine a single-rollout asynchronous recipe combining Muon, token-averaged loss, truncated importance sampling, and auxiliary length and tool-error penalties. Single-rollout sampling allows asynchronous learning without waiting for multiple trajectories for the same issue. We study optimizer choices and recipe components for repository-level issue resolution across two model families. The evaluated training runs show higher training success with the full recipe than with the baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 32 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

19 of 32 references.

References · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 336
  5. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
  6. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 158
  7. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practicescited by 41
  8. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learningcited by 37
  9. 2026SWE-World: Building Software Engineering Agents in Docker-Free Environmentscited by 25
  10. 2026SWE-Master: Unleashing the Potential of Software Engineering Agents via Post-Trainingcited by 22
  11. 2026ARLArena: A Unified Framework for Stable Agentic Reinforcement Learningcited by 13
  12. 2026Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMscited by 11
7 more
  1. 2025Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalizationcited by 9
  2. 2026Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipecited by 8
  3. 2026Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVRcited by 8
  4. 2026Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPOcited by 3
  5. 2026Reward Engineering for Software Tasks: A Survey of Reinforcement Learning Approachescited by 2
  6. 2026SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RLcited by 2
  7. 2026When Does Muon Help Agentic Reinforcement Learning?cited by 1

Built on by

Not cited yet.