mlsanityData to 14 Oct 2026

When Should We Prefer State-to-Visual DAgger Over Visual Reinforcement Learning?

arXiv 2412.13662 ↗First posted 18 Dec 2024cs.CV cs.AI cs.LGcited by 7

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

Learning policies from high-dimensional visual inputs, such as pixels and point clouds, is crucial in various applications. Visual reinforcement learning is a promising approach that directly trains policies from visual observations, although it faces challenges in sample efficiency and computational costs. This study conducts an empirical comparison of State-to-Visual DAgger, a two-stage framework that initially trains a state policy before adopting online imitation to learn a visual policy, and Visual RL across a diverse set of tasks. We evaluate both methods across 16 tasks from three benchmarks, focusing on their asymptotic performance, sample efficiency, and computational costs. Surprisingly, our findings reveal that State-to-Visual DAgger does not universally outperform Visual RL but shows significant advantages in challenging tasks, offering more consistent performance. In contrast, its benefits in sample efficiency are less pronounced, although it often reduces the overall wall-clock time required for training. Based on our findings, we provide recommendations for practitioners and hope that our results contribute valuable perspectives for future research in visual policy learning.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 32 of its 43 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

32 of 43 references.

References · 32

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  6. 2020Mastering Atari with Discrete World Modelscited by 1,007
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  9. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  11. 2018Learning Dexterous In-Hand Manipulationcited by 687
  12. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
20 more
  1. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  2. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  3. 2020Reinforcement Learning with Augmented Datacited by 468
  4. 2019Learning by Cheatingcited by 383
  5. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 382
  6. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327
  7. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  8. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 238
  9. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 197
  10. 2021A System for General In-Hand Object Re-Orientationcited by 165
  11. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  12. 2021Learning High-Speed Flight in the Wildcited by 128
  13. 2023UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policycited by 124
  14. 2022Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapescited by 101
  15. 2021RRL: Resnet as representation for Reinforcement Learningcited by 91
  16. 2019Does computer vision matter for action?cited by 64
  17. 2022Multi-skill Mobile Manipulation for Object Rearrangementcited by 46
  18. 2021Learning to Jump from Pixelscited by 45
  19. 2021Reinforcement Learning with Latent Flowcited by 19
  20. 2023Autonomous Driving with Deep Reinforcement Learning in CARLA Simulationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 21
  2. 2025Accelerating Visual-Policy Learning through Parallel Differentiable Simulationcited by 5
  3. 2025To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable Reinforcement Learningcited by 3
  4. 2026Squint: Fast Visual Reinforcement Learning for Sim-to-Real Roboticscited by 2
  5. 2026Task-Aware Exploration via a Predictive Bisimulation Metriccited by 1
  6. 2026Efficient On-policy Visual-RL via Stochastic Decoupled Policy Gradientcited by 1
  7. 2026Accelerating Visual Policy Learning with Sampling-Based Model Predictive Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Pretrained Visual Representations in Reinforcement Learning
  2. 2025Pre-trained Visual Representations Generalize Where it Matters in Model-Based Reinforcement Learning
  3. 2024The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning
  4. 2025Reinforcement Learning for Large Model: A Survey
  5. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations
  6. 2024MOOSS: Mask-Enhanced Temporal Contrastive Learning for Smooth State Evolution in Visual Reinforcement Learning
  7. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learning
  8. 2026Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Control