# Learning to Jump from Pixels

arXiv: https://arxiv.org/abs/2110.15344 · HTML: https://mlsanity.com/paper/2110.15344/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 28 Oct 2021.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Learning by Cheating](https://mlsanity.com/paper/1912.12294/) (2019)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning](https://mlsanity.com/paper/2105.08328/) (2021)
- [Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robots](https://mlsanity.com/paper/2111.01674/) (2021)
- [Learning and Transfer of Modulated Locomotor Controllers](https://mlsanity.com/paper/1610.05182/) (2016)
- [RLOC: Terrain-Aware Legged Locomotion using Reinforcement Learning and Optimal Control](https://mlsanity.com/paper/2012.03094/) (2020)
- [Highly Dynamic Quadruped Locomotion via Whole-Body Impulse Control and Model Predictive Control](https://mlsanity.com/paper/1909.06586/) (2019)
- [ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skills](https://mlsanity.com/paper/2005.04323/) (2020)
- [Fast and Efficient Locomotion via Learned Gait Transitions](https://mlsanity.com/paper/2104.04644/) (2021)
- [DeepGait: Planning and Control of Quadrupedal Gaits using Deep Reinforcement Learning](https://mlsanity.com/paper/1909.08399/) (2019)
- [Policies Modulating Trajectory Generators](https://mlsanity.com/paper/1910.02812/) (2019)
- [Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?](https://mlsanity.com/paper/1611.01055/) (2016)
- [Robust Quadruped Jumping via Deep Reinforcement Learning](https://mlsanity.com/paper/2011.07089/) (2020)
- [Multi-Layered Safety for Legged Robots via Control Barrier Functions and Model Predictive Control](https://mlsanity.com/paper/2011.00032/) (2020)
- [GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model](https://mlsanity.com/paper/2104.09771/) (2021)
- [Learning a Contact-Adaptive Controller for Robust, Efficient Legged Locomotion](https://mlsanity.com/paper/2009.10019/) (2020)
- [From Pixels to Legs: Hierarchical Learning of Quadruped Locomotion](https://mlsanity.com/paper/2011.11722/) (2020)
- [Fast and Continuous Foothold Adaptation for Dynamic Locomotion through CNNs](https://mlsanity.com/paper/1809.09759/) (2018)
- [Learning Agile Locomotion Skills with a Mentor](https://mlsanity.com/paper/2011.05541/) (2020)
- [Computationally-Robust and Efficient Prioritized Whole-Body Controller with Contact Constraints](https://mlsanity.com/paper/1807.01222/) (2018)

## Built on by (most cited first, 30 of 45)

- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [Rapid Locomotion via Reinforcement Learning](https://mlsanity.com/paper/2205.02824/) (2022)
- [Extreme Parkour with Legged Robots](https://mlsanity.com/paper/2309.14341/) (2023)
- [Legged Locomotion in Challenging Terrains using Egocentric Vision](https://mlsanity.com/paper/2211.07638/) (2022)
- [Robot Parkour Learning](https://mlsanity.com/paper/2309.05665/) (2023)
- [Expressive Whole-Body Control for Humanoid Robots](https://mlsanity.com/paper/2402.16796/) (2024)
- [Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior](https://mlsanity.com/paper/2212.03238/) (2022)
- [Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion](https://mlsanity.com/paper/2210.10044/) (2022)
- [Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapes](https://mlsanity.com/paper/2211.11744/) (2022)
- [AMO: Adaptive Motion Optimization for Hyper-Dexterous Humanoid Whole-Body Control](https://mlsanity.com/paper/2505.03738/) (2025)
- [BeamDojo: Learning Agile Humanoid Locomotion on Sparse Footholds](https://mlsanity.com/paper/2502.10363/) (2025)
- [Neural Volumetric Memory for Visual Locomotion Control](https://mlsanity.com/paper/2304.01201/) (2023)
- [Barkour: Benchmarking Animal-level Agility with Quadruped Robots](https://mlsanity.com/paper/2305.14654/) (2023)
- [Robust and Versatile Bipedal Jumping Control through Reinforcement Learning](https://mlsanity.com/paper/2302.09450/) (2023)
- [CPG-RL: Learning Central Pattern Generators for Quadruped Locomotion](https://mlsanity.com/paper/2211.00458/) (2022)
- [Learning and Adapting Agile Locomotion Skills by Transferring Experience](https://mlsanity.com/paper/2304.09834/) (2023)
- [Learning Torque Control for Quadrupedal Locomotion](https://mlsanity.com/paper/2203.05194/) (2022)
- [Learning Visual Locomotion with Cross-Modal Supervision](https://mlsanity.com/paper/2211.03785/) (2022)
- [A Survey on Physics Informed Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2309.01909/) (2023)
- [Grow Your Limits: Continuous Improvement with Real-World RL for Robotic Locomotion](https://mlsanity.com/paper/2310.17634/) (2023)
- [ViNL: Visual Navigation and Locomotion Over Obstacles](https://mlsanity.com/paper/2210.14791/) (2022)
- [Curriculum-Based Reinforcement Learning for Quadrupedal Jumping: A Reference-free Design](https://mlsanity.com/paper/2401.16337/) (2024)
- [TGRL: An Algorithm for Teacher Guided Reinforcement Learning](https://mlsanity.com/paper/2307.03186/) (2023)
- [Visual CPG-RL: Learning Central Pattern Generators for Visually-Guided Quadruped Locomotion](https://mlsanity.com/paper/2212.14400/) (2022)
- [SoloParkour: Constrained Reinforcement Learning for Visual Locomotion from Privileged Experience](https://mlsanity.com/paper/2409.13678/) (2024)
- [Solving Minimum-Cost Reach Avoid using Reinforcement Learning](https://mlsanity.com/paper/2410.22600/) (2024)
- [PI-ARS: Accelerating Evolution-Learned Visual-Locomotion with Predictive Information Representations](https://mlsanity.com/paper/2207.13224/) (2022)
- [Learning to See Physical Properties with Active Sensing Motor Policies](https://mlsanity.com/paper/2311.01405/) (2023)
- [Vegetable Peeling: A Case Study in Constrained Dexterous Manipulation](https://mlsanity.com/paper/2407.07884/) (2024)
- [Learning Robust, Agile, Natural Legged Locomotion Skills in the Wild](https://mlsanity.com/paper/2304.10888/) (2023)

## Abstract (arXiv)

Today's robotic quadruped systems can robustly walk over a diverse range of rough but continuous terrains, where the terrain elevation varies gradually. Locomotion on discontinuous terrains, such as those with gaps or obstacles, presents a complementary set of challenges. In discontinuous settings, it becomes necessary to plan ahead using visual inputs and to execute agile behaviors beyond robust walking, such as jumps. Such dynamic motion results in significant motion of onboard sensors, which introduces a new set of challenges for real-time visual processing. The requirement for agility and terrain awareness in this setting reinforces the need for robust control. We present Depth-based Impulse Control (DIC), a method for synthesizing highly agile visually-guided locomotion behaviors. DIC affords the flexibility of model-free learning but regularizes behavior through explicit model-based optimization of ground reaction forces. We evaluate the proposed method both in simulation and in the real world.