VersualRL: Closed-Loop Verbal Reinforcement Learning with Visual Execution Feedback for Task-Level Robot Planning
On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation
Abstract (arXiv)
We introduce VersualRL, a closed-loop framework for task-level robot planning that uses visual execution feedback to iteratively refine executable Behavior Trees through structured natural-language critique. VersualRL combines a vision-language model critic, which analyzes visual observations and Behavior Tree execution traces, with a large language model actor that performs discrete and interpretable policy updates. During physical deployment, both foundation models remain frozen, while the executed Behavior Tree is updated at the symbolic level without online gradient-based policy optimization. This enables transparent reasoning, explicit causal feedback, and human-interpretable policy evolution. We validate VersualRL on a real mobile robot performing a multi-stage navigation and manipulation task under execution uncertainty. Experimental results show that the framework supports explainable policy improvements, closed-loop adaptation to execution failures, and successful deployment on physical robotic systems.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 25 of its 32 references that are in the map. First posted on arXiv 23 Mar 2026.
Builds on
25 of 32 references.
Closest ancestors · 1
Same direction, newest first
Other references · 24
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
- 2016Concrete Problems in AI Safetycited by 1,642
- 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
- 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
- 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
- 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 165
- 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 156
- 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 141
12 more
- 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 89
- 2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 25
- 2023LLM-BRAIn: AI-driven Fast Generation of Robot Behaviour Tree based on Large Language Modelcited by 24
- 2024VLM-Auto: VLM-based Autonomous Driving Assistant with Human-like Behavior and Understanding for Complex Road Scenescited by 23
- 2024Learning Reward for Robot Skills Using Large Language Models via Self-Alignmentcited by 20
- 2024AutoGPT+P: Affordance-based Task Planning with Large Language Modelscited by 19
- 2023LLM-MARS: Large Language Model for Behavior Tree Generation and NLP-enhanced Dialogue in Multi-Agent Robot Systemscited by 18
- 2024BTGenBot: Behavior Tree Generation for Robotic Tasks with Lightweight LLMscited by 14
- 2023CLIP-Motion: Learning Reward Functions for Robotic Actions Using Consecutive Observationscited by 9
- 2025VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferencescited by 9
- 2025HIPPO-MAT: Decentralized Task Allocation Using GraphSAGE and Multi-Agent Deep Reinforcement Learningcited by 3
- 2025AgilePilot: DRL-Based Drone Agent for Real-Time Motion Planning in Dynamic Environments by Leveraging Object Detectioncited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Vision-Language-Policy Model for Dynamic Robot Task Planning
- 2025ExploreVLM: Closed-Loop Robot Exploration Task Planning with Vision-Language Models
- 2026PAPO-VLA: Planning-Aware Policy Optimization for Vision-Language-Action Models
- 2025TRACE: A Self-Improving Framework for Robot Behavior Forecasting with Vision-Language Models
- 2026VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents
- 2025Reinforced Embodied Planning with Verifiable Reward for Real-World Robotic Manipulation
- 2025Using Language Models as Closed-Loop High-Level Planners for Robotics Applications: A Brief Overview and Benchmarks
- 2026A Few Words Go a Long Way: Language Guided Robot Policy Synthesis