mlsanityData to 14 Oct 2026

Visual Whole-Body Control for Legged Loco-Manipulation

arXiv 2403.16967 ↗First posted 25 Mar 2024cs.RO cs.CV cs.LGcited by 62

Not on the map. Why?

Cited by 20 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study the problem of mobile manipulation using legged robots equipped with an arm, namely legged loco-manipulation. The robot legs, while usually utilized for mobility, offer an opportunity to amplify the manipulation capabilities by conducting whole-body control. That is, the robot can control the legs and the arm at the same time to extend its workspace. We propose a framework that can conduct the whole-body control autonomously with visual observations. Our approach, namely Visual Whole-Body Control(VBC), is composed of a low-level policy using all degrees of freedom to track the body velocities along with the end-effector position, and a high-level policy proposing the velocities and end-effector position based on visual inputs. We train both levels of policies in simulation and perform Sim2Real transfer for real robot deployment. We perform extensive experiments and show significant improvements over baselines in picking up diverse objects in different configurations (heights, locations, orientations) and environments.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Builds on 57 of its 91 references that are in the map. First posted on arXiv 25 Mar 2024.

Builds on

57 of 91 references.

References · 57

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Segment Anythingcited by 8,234
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  6. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  7. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  8. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  9. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  10. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 391
  11. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 382
  12. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
38 more
  1. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 251
  2. 2021Associating Objects with Transformers for Video Object Segmentationcited by 226
  3. 2023Extreme Parkour with Legged Robotscited by 164
  4. 2022Legged Locomotion in Challenging Terrains using Egocentric Visioncited by 151
  5. 2018PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planningcited by 150
  6. 2023Robot Parkour Learningcited by 149
  7. 2022Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behaviorcited by 138
  8. 2022Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotioncited by 121
  9. 2021Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robotscited by 116
  10. 2023On Bringing Robots Homecited by 101
  11. 2022Adversarial Motion Priors Make Good Substitutes for Complex Reward Functionscited by 89
  12. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
  13. 2021Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulationcited by 59
  14. 2022Adapting Rapid Motor Adaptation for Bipedal Robotscited by 54
  15. 2021Coupling Vision and Proprioception for Navigation of Legged Robotscited by 51
  16. 2020ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulationcited by 49
  17. 2022Multi-skill Mobile Manipulation for Object Rearrangementcited by 46
  18. 2022Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrationscited by 45
  19. 2023Neural Volumetric Memory for Visual Locomotion Controlcited by 44
  20. 2023ASC: Adaptive Skill Coordination for Robotic Mobile Manipulationcited by 42
  21. 2023Learning Vision-Based Bipedal Locomotion for Challenging Terraincited by 42
  22. 2020Energy-Based Imitation Learningcited by 40
  23. 2023Legs as Manipulator: Pushing Quadrupedal Agility Beyond Locomotioncited by 39
  24. 2022Combining Learning-based Locomotion Policy with Model-based Manipulation for Legged Mobile Manipulatorscited by 37
  25. 2021Fully Autonomous Real-World Reinforcement Learning with Applications to Mobile Manipulationcited by 35
  26. 2022OPT-Mimic: Imitation of Optimized Trajectories for Dynamic Quadruped Behaviorscited by 35
  27. 2023Versatile Multi-Contact Planning and Control for Legged Loco-Manipulationcited by 31
  28. 2021VILENS: Visual, Inertial, Lidar, and Leg Odometry for All-Terrain Legged Robotscited by 25
  29. 2023Learning Whole-body Manipulation for Quadrupedal Robotcited by 25
  30. 2023GAMMA: Graspability-Aware Mobile MAnipulation Policy Learning based on Online Grasping Pose Fusioncited by 22
  31. 2023Generalized Animal Imitator: Agile Locomotion with Versatile Motion Priorcited by 19
  32. 2021Learning Inertial Odometry for Dynamic Legged Robot State Estimationcited by 17
  33. 2023Harmonic Mobile Manipulationcited by 17
  34. 2022RoLoMa: Robust Loco-Manipulation for Quadruped Robots with Armscited by 16
  35. 2022Bayesian Multi-Task Learning MPC for Robotic Mobile Manipulationcited by 16
  36. 2017Deep Kernels for Optimizing Locomotion Controllerscited by 14
  37. 2022Reinforcement Learning with Automated Auxiliary Loss Searchcited by 13
  38. 2022Continuous Jumping for Legged Robots on Stepping Stones via Trajectory Optimization and Model Predictive Controlcited by 10

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 16; 2025: 31; 2026: 15.

62 papers.

Sort:

By area:

  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  2. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 176
  3. 2025Humanoid Policy ~ Human Policycited by 78
  4. 2025HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpitcited by 76
  5. 2024GRUtopia: Dream General Robots in a City at Scalecited by 59
  6. 2024Mobile-TeleVision: Predictive Motion Priors for Humanoid Whole-Body Controlcited by 51
  7. 2025Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoidscited by 48
  8. 2025LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instructioncited by 43
  9. 2024GraspSplats: Efficient Manipulation with 3D Feature Splattingcited by 41
  10. 2025BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activitiescited by 31
  11. 2024Learning Smooth Humanoid Locomotion through Lipschitz-Constrained Policiescited by 30
  12. 2024M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenescited by 27
38 more
  1. 2025Learning coordinated badminton skills for legged manipulatorscited by 19
  2. 2025PhysHSI: Towards a Real-World Generalizable and Natural Humanoid-Scene Interaction Systemcited by 18
  3. 2025Towards Autonomous Micromobility through Scalable Urban Simulationcited by 16
  4. 2025Towards Embodiment Scaling Laws in Robot Locomotioncited by 16
  5. 2025Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfercited by 15
  6. 2024Learning Multi-Agent Loco-Manipulation for Long-Horizon Quadrupedal Pushingcited by 13
  7. 2025Bridging the Sim-to-Real Gap for Athletic Loco-Manipulationcited by 13
  8. 2025A Unified and General Humanoid Whole-Body Controller for Versatile Locomotioncited by 11
  9. 2025FACET: Force-Adaptive Control via Impedance Reference Tracking for Legged Robotscited by 11
  10. 2025Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robotscited by 11
  11. 2025UMI-on-Air: Embodiment-Aware Guidance for Embodiment-Agnostic Visuomotor Policiescited by 11
  12. 2024Guided Reinforcement Learning for Robust Multi-Contact Loco-Manipulationcited by 10
  13. 2024Learning Whole-Body Loco-Manipulation for Omni-Directional Task Space Pose Tracking with a Wheeled-Quadrupedal-Manipulatorcited by 10
  14. 2025Human2LocoMan: Learning Versatile Quadrupedal Manipulation with Human Pretrainingcited by 9
  15. 2025ODYSSEY: Open-World Quadrupeds Exploration and Manipulation for Long-Horizon Taskscited by 9
  16. 2024Helpful DoggyBot: Open-World Object Fetching using Legged Robots and Vision-Language Modelscited by 8
  17. 2024Whole-body End-Effector Pose Trackingcited by 6
  18. 2024ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimizationcited by 6
  19. 2024WildLMa: Long Horizon Loco-Manipulation in the Wildcited by 6
  20. 2025Dynamic object goal pushing with mobile manipulators through model-free constrained reinforcement learningcited by 5
  21. 2025DreamPolicy: A Unified World-model Policy for Scalable Humanoid Locomotioncited by 5
  22. 2025Versatile Loco-Manipulation through Flexible Interlimb Coordinationcited by 5
  23. 2026OpenHLM: An Empirical Recipe for Whole-Body Humanoid Loco-Manipulationcited by 5
  24. 2025SLIM: Sim-to-Real Legged Instructive Manipulation via Long-Horizon Visuomotor Learningcited by 4
  25. 2025Optimistic critics can empower small actorscited by 4
  26. 2024Bimanual Grasp Synthesis for Dexterous Robot Handscited by 3
  27. 2025Spectral Normalization for Lipschitz-Constrained Policies on Learning Humanoid Locomotioncited by 3
  28. 2025Bipedalism for Quadrupedal Robots: Versatile Loco-Manipulation through Risk-Adaptive Reinforcement Learningcited by 3
  29. 2026StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perceptioncited by 3
  30. 2026SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Frameworkcited by 3
  31. 2025Unreal Robotics Lab: A High-Fidelity Robotics Simulator with Advanced Physics and Renderingcited by 2
  32. 2025Distilling Realizable Students from Unrealizable Teacherscited by 2
  33. 2025EBT-Policy: Energy Unlocks Emergent Physical Reasoning Capabilitiescited by 2
  34. 2026Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulationcited by 2
  35. 2024QuadWBG: Generalizable Quadrupedal Whole-Body Graspingcited by 1
  36. 2025MASH: Cooperative-Heterogeneous Multi-Agent Reinforcement Learning for Single Humanoid Robot Locomotioncited by 1
  37. 2025SAGA: Open-World Mobile Manipulation via Structured Affordance Groundingcited by 1
  38. 2026HERO: Learning Humanoid End-Effector Control for Visual Whole-Body Open-Vocabulary Object Graspingcited by 1

50 most cited of 62 All 62 (JSON)