mlsanityData to 14 Oct 2026

OpenClaw-RL: Train Any Agent Simply by Talking

arXiv 2603.10165 ↗First posted 10 Mar 2026cs.CL cs.AI cs.CVcited by 90

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Every agent interaction generates a next-state signal, namely the user reply, tool output, terminal or GUI state change that follows each action, yet no existing agentic RL system recovers it as a live, online learning source. We present OpenClaw-RL, a framework that employs next-state signals to optimize personal agents online through infrastructure and methodology innovations. On the infrastructure side, we extend existing RL systems to a server-client architecture where the RL server hosts the policy behind an inference API and user terminals stream interaction data back over HTTP. From each observed next state, the system extracts two complementary training signals, evaluative and directive, via a separate asynchronous server so that neither signal extraction nor optimization blocks inference. On the methodology side, we introduce a hybrid RL objective that unifies both signal types in a single update: directive signals provide richer, token-level supervision but are sparser, while evaluative signals are more broadly available. To stabilize distillation under teacher-student mismatch, we propose overlap-guided hint selection, which picks the hint whose induced teacher distribution maximally overlaps with the student's top-k tokens, together with a log-probability-difference clip that bounds per-token advantages. Applied to personal agents, OpenClaw-RL enables an agent to improve simply by being used, recovering conversational signals from user re-queries, corrections, and explicit feedback. Applied to general agents, OpenClaw-RL is the first RL framework to unify real-world agent settings spanning terminal, GUI, SWE, and tool-call environments, where we additionally demonstrate the utility of next-state signals in long-horizon settings.

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 56 of its 85 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

56 of 85 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System2 Feb
  2. 2025Demystifying Reinforcement Learning in Agentic Reasoning13 Oct
  3. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  4. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMs15 Apr
  5. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agents3 Feb
  6. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL29 Feb

Other references · 50

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  12. 2025Qwen3-VL Technical Reportcited by 2,793
38 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  2. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  3. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  4. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  5. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  6. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  7. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  8. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  9. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  10. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
  11. 2024Self-Rewarding Language Modelscited by 681
  12. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  13. 2025Process Reinforcement through Implicit Rewardscited by 504
  14. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  15. 2025Zep: A Temporal Knowledge Graph Architecture for Agent Memorycited by 400
  16. 2026Reinforcement Learning via Self-Distillationcited by 332
  17. 2023FireAct: Toward Language Agent Fine-tuningcited by 256
  18. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  19. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  20. 2026Self-Distillation Enables Continual Learningcited by 247
  21. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 215
  22. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  23. 2025MIRIX: Multi-Agent Memory System for LLM-Based Agentscited by 173
  24. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 160
  25. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 101
  26. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 97
  27. 2024Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Modelscited by 91
  28. 2026Qwen3-Coder-Next Technical Reportcited by 71
  29. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 63
  30. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 62
  31. 2025Co-Evolving LLM Coder and Unit Tester via Reinforcement Learningcited by 56
  32. 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 53
  33. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 46
  34. 2024SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correctioncited by 40
  35. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 37
  36. 2026Aligning Language Models from User Interactionscited by 28
  37. 2026Experiential Reinforcement Learningcited by 22
  38. 2025Optimizing the Interface Between Knowledge Graphs and LLMs for Complex Reasoningcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 90.

90 papers.

Sort:

By area:

  1. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
  2. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 145
  3. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 90
  4. 2026SkillClaw: Let Skills Evolve Collectively with Agentic Evolvercited by 81
  5. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 76
  6. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 49
  7. 2026TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agentscited by 47
  8. 2026Continual Harness: Online Adaptation for Self-Improving Foundation Agentscited by 41
  9. 2026Dynamic Dual-Granularity Skill Bank for Agentic RLcited by 34
  10. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 21
  11. 2026ECHO: Terminal Agents Learn World Models for Freecited by 19
  12. 2026SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolutioncited by 18
38 more
  1. 2026GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversationscited by 17
  2. 2026Are Full Rollouts Necessary for On-Policy Distillation?cited by 17
  3. 2026ClawGym: A Scalable Framework for Building Effective Claw Agentscited by 15
  4. 2026SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillationcited by 15
  5. 2026ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agentscited by 14
  6. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularizationcited by 12
  7. 2026ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agentscited by 11
  8. 2026Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVRcited by 11
  9. 2026HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agentscited by 10
  10. 2026The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillationcited by 9
  11. 2026PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agentscited by 9
  12. 2026ClawEnvKit: Automatic Environment Generation for Claw-Like Agentscited by 8
  13. 2026π-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Datacited by 7
  14. 2026ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactionscited by 7
  15. 2026OpenForgeRL: Train Harness-native Agents in Any Environmentcited by 7
  16. 2026ScienceBuddy: Recursive-in-Recursive Self-Improvement for Interactive Scientific Agentscited by 7
  17. 2026OPSD Compresses What RLVR Teaches: A Post-RL Compaction Stage for Reasoning Modelscited by 6
  18. 2026Bridging the Agent-World Gap: Text World Models for LLM-based Agentscited by 5
  19. 2026Xiaomi-GUI-0 Technical Reportcited by 5
  20. 2026From Proprietary to Open-Source: Bridging the Distribution Gap via Multi-Agent Protocol Distillation in Agentic Searchcited by 5
  21. 2026DREvo: Distilling Recalibrated Historical Experience for Harness Self-Evolutioncited by 5
  22. 2026DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Trainingcited by 5
  23. 2026RISE: Recursive Improvement via Self-Extrapolating Policy Distillationcited by 5
  24. 2026Learning Agentic Policy from Action Guidancecited by 4
  25. 2026BraveGuard: From Open-World Threats to Safer Computer-Use Agentscited by 4
  26. 2026The Role of Feedback Alignment in Self-Distillationcited by 4
  27. 2026LLM-as-a-Coach: Experiential Learning for Non-Verifiable Taskscited by 4
  28. 2026Meta-Task: Turning Terminal Task Synthesis into a Terminal Task for Scalable Agent Trainingcited by 4
  29. 2026RewardFlow: Topology-Aware Reward Propagation on State Graphs for Agentic RL with Large Language Modelscited by 3
  30. 2026Learning to Commit: Generating Organic Pull Requests via Online Repository Memorycited by 3
  31. 2026Federation over Text: Insight Sharing for Multi-Agent Reasoningcited by 3
  32. 2026OpenJarvis: Personal AI, On Personal Devicescited by 3
  33. 2026TimeClaw: A Time-Series AI Agent with Exploratory Execution Learningcited by 2
  34. 2026Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Valuescited by 2
  35. 2026EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearingcited by 2
  36. 2026Statistical Priors for Implicit Preferences: Decoupling Skill Selection as a Local Harness in Personal Agentscited by 2
  37. 2026OpenClaw-Skill: Collective Skill Tree Search for Agentic Large Language Modelscited by 2
  38. 2026Med-OPD: Improving Medical Vision-Language Models via Evidence-Aware On-Policy Distillationcited by 2

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Action-Free Offline-to-Online RL via Discretised State Policies
  2. 2026MetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wild
  3. 2026From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents
  4. 2026OpenThoughts-Agent: Data Recipes for Agentic Models
  5. 2026OpenTinker: Separating Concerns in Agentic Reinforcement Learning
  6. 2026WEFT: Scaling Tool-Use Post-Training for General-Purpose Agents
  7. 2026On Effectiveness and Efficiency of Agentic Tool-calling and RL Training
  8. 2025JEF-Hinter: Leveraging Offline Knowledge for Improving Web Agents Adaptation