# RobotKeyframing: Learning Locomotion with High-Level Objectives via Mixture of Dense and Sparse Rewards

arXiv: https://arxiv.org/abs/2407.11562 · HTML: https://mlsanity.com/paper/2407.11562/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 53 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Robust Motion In-betweening](https://mlsanity.com/paper/2102.04942/) (2021)
- [Character Controllers Using Motion VAEs](https://mlsanity.com/paper/2103.14274/) (2021)
- [Extreme Parkour with Legged Robots](https://mlsanity.com/paper/2309.14341/) (2023)
- [ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robots](https://mlsanity.com/paper/2306.14874/) (2023)
- [Legged Locomotion in Challenging Terrains using Egocentric Vision](https://mlsanity.com/paper/2211.07638/) (2022)
- [Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior](https://mlsanity.com/paper/2212.03238/) (2022)
- [Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions](https://mlsanity.com/paper/2203.15103/) (2022)
- [Humanoid Locomotion as Next Token Prediction](https://mlsanity.com/paper/2402.19469/) (2024)
- [Advanced Skills by Learning Locomotion and Local Navigation End-to-End](https://mlsanity.com/paper/2209.12827/) (2022)
- [DiffuseLoco: Real-Time Legged Locomotion Control with Diffusion from Offline Datasets](https://mlsanity.com/paper/2404.19264/) (2024)
- [Learning Robust Autonomous Navigation and Locomotion for Wheeled-Legged Robots](https://mlsanity.com/paper/2405.01792/) (2024)
- [Lifelike Agility and Play in Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models](https://mlsanity.com/paper/2308.15143/) (2023)
- [Whole-body Humanoid Robot Locomotion with Human Reference](https://mlsanity.com/paper/2402.18294/) (2024)
- [Composite Motion Learning with Task Control](https://mlsanity.com/paper/2305.03286/) (2023)
- [Pedipulate: Enabling Manipulation Skills using a Quadruped Robot's Leg](https://mlsanity.com/paper/2402.10837/) (2024)

## Built on by (most cited first, 18 of 18)

- [Learning Humanoid Standing-up Control across Diverse Postures](https://mlsanity.com/paper/2502.08378/) (2025)
- [BeamDojo: Learning Agile Humanoid Locomotion on Sparse Footholds](https://mlsanity.com/paper/2502.10363/) (2025)
- [A Learning Framework for Diverse Legged Robot Locomotion Using Barrier-Based Style Rewards](https://mlsanity.com/paper/2409.15780/) (2024)
- [Towards Adaptable Humanoid Control via Adaptive Motion Tracking](https://mlsanity.com/paper/2510.14454/) (2025)
- [Embrace Collisions: Humanoid Shadowing for Deployable Contact-Agnostics Motions](https://mlsanity.com/paper/2502.01465/) (2025)
- [DreamPolicy: A Unified World-model Policy for Scalable Humanoid Locomotion](https://mlsanity.com/paper/2505.18780/) (2025)
- [Non-conflicting Energy Minimization in Reinforcement Learning based Robot Control](https://mlsanity.com/paper/2509.01765/) (2025)
- [FlowCritic: Bridging Value Estimation with Flow Matching in Reinforcement Learning](https://mlsanity.com/paper/2510.22686/) (2025)
- [Robot Crash Course: Learning Soft and Stylized Falling](https://mlsanity.com/paper/2511.10635/) (2025)
- [Bipedalism for Quadrupedal Robots: Versatile Loco-Manipulation through Risk-Adaptive Reinforcement Learning](https://mlsanity.com/paper/2507.20382/) (2025)
- [Feature-Based vs. GAN-Based Learning from Demonstrations: When and Why](https://mlsanity.com/paper/2507.05906/) (2025)
- [Arnold: A multi-task, multi-embodiment muscle transformer policy](https://mlsanity.com/paper/2508.18066/) (2025)
- [LIPM-Guided Reinforcement Learning for Stable and Perceptive Locomotion in Bipedal Robots](https://mlsanity.com/paper/2509.09106/) (2025)
- [PUMA: Perception-driven Unified Foothold Prior for Mobility Augmented Quadruped Parkour](https://mlsanity.com/paper/2601.15995/) (2026)
- [LineRides: Line-Guided Reinforcement Learning for Bicycle Robot Stunts](https://mlsanity.com/paper/2605.05110/) (2026)
- [KiRAS: Keyframe Guided Self-Imitation for Robust and Adaptive Skill Learning in Quadruped Robots](https://mlsanity.com/paper/2603.15179/) (2026)
- [Critic Architecture Matters: Dual vs. Unified Critics for Humanoid Loco-Manipulation](https://mlsanity.com/paper/2606.11891/) (2026)
- [Bicycle Acrobatics with Reinforcement Learning](https://mlsanity.com/paper/2608.00880/) (2026)

## Abstract (arXiv)

This paper presents a novel learning-based control framework that uses keyframing to incorporate high-level objectives in natural locomotion for legged robots. These high-level objectives are specified as a variable number of partial or complete pose targets that are spaced arbitrarily in time. Our proposed framework utilizes a multi-critic reinforcement learning algorithm to effectively handle the mixture of dense and sparse rewards. Additionally, it employs a transformer-based encoder to accommodate a variable number of input targets, each associated with specific time-to-arrivals. Throughout simulation and hardware experiments, we demonstrate that our framework can effectively satisfy the target keyframe sequence at the required times. In the experiments, the multi-critic method significantly reduces the effort of hyperparameter tuning compared to the standard single-critic alternative. Moreover, the proposed transformer-based architecture enables robots to anticipate future goals, which results in quantitative improvements in their ability to reach their targets.