mlsanityData to 14 Oct 2026

Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers

arXiv 2107.03996 ↗First posted 8 Jul 2021cs.LG cs.CV cs.ROcited by 79

Not on the map. Why?

Cited by 20 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose to address quadrupedal locomotion tasks using Reinforcement Learning (RL) with a Transformer-based model that learns to combine proprioceptive information and high-dimensional depth sensor inputs. While learning-based locomotion has made great advances using RL, most methods still rely on domain randomization for training blind agents that generalize to challenging terrains. Our key insight is that proprioceptive states only offer contact measurements for immediate reaction, whereas an agent equipped with visual sensory observations can learn to proactively maneuver environments with obstacles and uneven terrain by anticipating changes in the environment many steps ahead. In this paper, we introduce LocoTransformer, an end-to-end RL method that leverages both proprioceptive states and visual observations for locomotion control. We evaluate our method in challenging simulated environments with different obstacles and uneven terrain. We transfer our learned policy from simulation to a real robot by running it indoors and in the wild with unseen obstacles and terrain. Our method not only significantly improves over baselines, but also achieves far better generalization performance, especially when transferred to the real robot. Our project page with videos is at https://rchalyang.github.io/LocoTransformer/ .

Key facts. Cited by 79 papers in the map (data through 14 Oct 2026). Builds on 58 of its 89 references that are in the map. First posted on arXiv 8 Jul 2021.

Builds on

58 of 89 references.

References · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2017Non-local Neural Networkscited by 2,293
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  10. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
38 more
  1. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  2. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  4. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  6. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  7. 2018Image Transformercited by 772
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  9. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  10. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  11. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  12. 2019Learning agile and dynamic motor skills for legged robotscited by 504
  13. 2020Reinforcement Learning with Augmented Datacited by 468
  14. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  15. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  16. 2020Multi-modal Transformer for Video Retrievalcited by 395
  17. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 377
  18. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 319
  19. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  20. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 316
  21. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  22. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  23. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 251
  24. 2017Cognitive Mapping and Planning for Visual Navigationcited by 212
  25. 2016Learning to Navigate in Complex Environmentscited by 198
  26. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  27. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  28. 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 165
  29. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  30. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 137
  31. 2018Visual Semantic Navigation using Scene Priorscited by 128
  32. 2018More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touchcited by 102
  33. 2020BADGR: An Autonomous Self-Supervised Learning-Based Navigation Systemcited by 102
  34. 2019Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Taskscited by 84
  35. 2021Decoupling the Role of Data, Attention, and Losses in Multimodal Transformerscited by 64
  36. 2018Learning Symmetric and Low-energy Locomotioncited by 61
  37. 2020Dynamics Randomization Revisited:A Case Study for Quadrupedal Locomotioncited by 59
  38. 2021Fast and Efficient Locomotion via Learned Gait Transitionscited by 54

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 19; 2023: 18; 2024: 17; 2025: 17; 2026: 6.

79 papers.

Sort:

By area:

  1. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  2. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  3. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 225
  4. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 211
  5. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 176
  6. 2023Extreme Parkour with Legged Robotscited by 164
  7. 2022Legged Locomotion in Challenging Terrains using Egocentric Visioncited by 151
  8. 2023Robot Parkour Learningcited by 149
  9. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 137
  10. 2022Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotioncited by 121
  11. 2023General In-Hand Object Rotation with Vision and Touchcited by 99
  12. 2024Agile But Safe: Learning Collision-Free High-Speed Legged Locomotioncited by 73
38 more
  1. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robotscited by 56
  2. 2021Coupling Vision and Proprioception for Navigation of Legged Robotscited by 51
  3. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  4. 2024Learning Robust Autonomous Navigation and Locomotion for Wheeled-Legged Robotscited by 48
  5. 2023Neural Volumetric Memory for Visual Locomotion Controlcited by 44
  6. 2023Hybrid Internal Model: Learning Agile Legged Locomotion with Simulated Robot Responsecited by 39
  7. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 36
  8. 2022CPG-RL: Learning Central Pattern Generators for Quadruped Locomotioncited by 34
  9. 2022Learning Torque Control for Quadrupedal Locomotioncited by 29
  10. 2024GeRM: A Generalist Robotic Model with Mixture-of-experts for Quadruped Robotcited by 29
  11. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 26
  12. 2022Learning Visual Locomotion with Cross-Modal Supervisioncited by 25
  13. 2021Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomizationcited by 23
  14. 2023Generalized Animal Imitator: Agile Locomotion with Versatile Motion Priorcited by 19
  15. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  16. 2023MoVie: Visual Model-Based Policy Adaptation for View Generalizationcited by 16
  17. 2025Towards Embodiment Scaling Laws in Robot Locomotioncited by 16
  18. 2023Contrastive Learning for Enhancing Robust Scene Transfer in Vision-based Agile Flightcited by 15
  19. 2024Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Modelscited by 15
  20. 2022Sim-to-Real Transfer for Quadrupedal Locomotion via Terrain Transformercited by 14
  21. 2022Visual CPG-RL: Learning Central Pattern Generators for Visually-Guided Quadruped Locomotioncited by 14
  22. 2022Neural Scene Representation for Locomotion on Structured Terraincited by 12
  23. 2023Generalizable Visual Reinforcement Learning with Segment Anything Modelcited by 12
  24. 2025GLEAM: Learning Generalizable Exploration Policy for Active Mapping in Complex 3D Indoor Scenescited by 11
  25. 2025Spatially-Enhanced Recurrent Memory for Long-Range Mapless Navigation via End-to-End Reinforcement Learningcited by 11
  26. 2023Physics-Regulated Deep Reinforcement Learning: Invariant Embeddingscited by 10
  27. 2023Learning to See Physical Properties with Active Sensing Motor Policiescited by 10
  28. 2025Distillation-PPO: A Novel Two-Stage Reinforcement Learning Framework for Humanoid Robot Perceptive Locomotioncited by 10
  29. 2024One Filter to Deploy Them All: Robust Safety for Quadrupedal Navigation in Unknown Environmentscited by 9
  30. 2022Learning to Walk by Steering: Perceptive Quadrupedal Locomotion in Dynamic Environmentscited by 8
  31. 2023Learning Robust, Agile, Natural Legged Locomotion Skills in the Wildcited by 8
  32. 2023Learning Sampling Dictionaries for Efficient and Generalizable Robot Motion Planning with Transformerscited by 8
  33. 2024Cross-Modal Learning for Anomaly Detection in Complex Industrial Process: Methodology and Benchmarkcited by 8
  34. 2022Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularizationcited by 7
  35. 2024SLR: Learning Quadruped Locomotion without Privileged Informationcited by 6
  36. 2024DreamWaQ++: Obstacle-Aware Quadrupedal Locomotion With Resilient Multi-Modal Reinforcement Learningcited by 6
  37. 2025Impoola: The Power of Average Pooling for Image-Based Deep Reinforcement Learningcited by 6
  38. 2022Learning Semantics-Aware Locomotion Skills from Human Demonstrationcited by 5

50 most cited of 79 All 79 (JSON)