mlsanityData to 7 Oct 2026

DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning

arXiv 2406.11896 ↗First posted 14 Jun 2024cs.LGcited by 210

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 2 more
Abstract (arXiv)

Training corpuses for vision language models (VLMs) typically lack sufficient amounts of decision-centric data. This renders off-the-shelf VLMs sub-optimal for decision-making tasks such as in-the-wild device control through graphical user interfaces (GUIs). While training with static demonstrations has shown some promise, we show that such methods fall short for controlling real GUIs due to their failure to deal with real-world stochasticity and non-stationarity not captured in static observational data. This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents through fine-tuning a pre-trained VLM in two stages: offline RL to initialize the model, followed by offline-to-online RL. To do this, we build a scalable and parallelizable Android learning environment equipped with a VLM-based evaluator and develop a simple yet effective RL approach for learning in this domain. Our approach runs advantage-weighted RL with advantage estimators enhanced to account for stochasticity along with an automatic curriculum for deriving maximal learning signal. We demonstrate the effectiveness of DigiRL using the Android-in-the-Wild (AitW) dataset, where our 1.3B VLM trained with RL achieves a 49.5% absolute improvement -- from 17.7 to 67.2% success rate -- over supervised fine-tuning with static human demonstration data. These results significantly surpass not only the prior best agents, including AppAgent with GPT-4V (8.3% success rate) and the 17B CogAgent trained with AitW data (38.5%), but also the prior best autonomous RL approach based on filtered behavior cloning (57.8%), thereby establishing a new state-of-the-art for digital agents for in-the-wild device control.

Key facts. Cited by 210 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 48 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

33 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning16 May
  2. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  9. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  10. 2015Prioritized Experience Replaycited by 1,070
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
19 more
  1. 2019Solving Rubik's Cube with a Robot Handcited by 699
  2. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  3. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  4. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  5. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  6. 2020Critic Regularized Regressioncited by 269
  7. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  8. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  9. 2020Prioritized Level Replaycited by 190
  10. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  11. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  12. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  13. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  14. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  15. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
  16. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  17. 2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
  18. 2024Benchmarking Mobile Device Control Agents across Diverse Configurationscited by 54
  19. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 98; 2026: 92.

210 papers.

Sort:

By area:

  1. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  2. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  3. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  4. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
  5. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 190
  6. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 182
  7. 2025Agentic Reinforced Policy Optimizationcited by 166
  8. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  9. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  10. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
  11. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  12. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
38 more
  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 103
  3. 2024GUI Agents: A Surveycited by 101
  4. 2024AutoGLM: Autonomous Foundation Agents for GUIscited by 96
  5. 2025Reinforcement Learning for Self-Improving Agent with Skill Librarycited by 91
  6. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  7. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 83
  8. 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systemscited by 79
  9. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluationcited by 79
  10. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  11. 2025PyVision: Agentic Vision with Dynamic Toolingcited by 71
  12. 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agentscited by 68
  13. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  14. 2024MindSearch: Mimicking Human Minds Elicits Deep AI Searchercited by 65
  15. 2025Self-Challenging Language Model Agentscited by 56
  16. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 55
  17. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  18. 2024The Limits of Inference Scaling Through Resamplingcited by 51
  19. 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 51
  20. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 45
  21. 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Userscited by 44
  22. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  23. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  24. 2025Fully Autonomous AI Agents Should Not be Developedcited by 43
  25. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
  26. 2025Web-Shepherd: Advancing PRMs for Reinforcing Web Agentscited by 41
  27. 2025Agentic Entropy-Balanced Policy Optimizationcited by 39
  28. 2025CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigationcited by 35
  29. 2025GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agentcited by 35
  30. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  31. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  32. 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
  33. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  34. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  35. 2025Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Modelscited by 30
  36. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 29
  37. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  38. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Surveycited by 28

50 most cited of 210 All 210 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving Vision-Language-Action Model with Online Reinforcement Learning
  2. 2025RLS3: RL-Based Synthetic Sample Selection to Enhance Spatial Reasoning in Vision-Language Models for Indoor Autonomous Perception
  3. 2025RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models
  4. 2025GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning
  5. 2026RL-VLA3: A Flexible and Asynchronous Reinforcement Learning Framework for VLA Training
  6. 2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation
  7. 2026Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models
  8. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics