# Video2Reward: Generating Reward Function from Videos for Legged Robot Behavior Learning

arXiv: https://arxiv.org/abs/2412.05515 · HTML: https://mlsanity.com/paper/2412.05515/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 7 Dec 2024.

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [DART: Noise Injection for Robust Imitation Learning](https://mlsanity.com/paper/1703.09327/) (2017)
- [A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms](https://mlsanity.com/paper/1907.03146/) (2019)
- [Learning Delicate Local Representations for Multi-Person Pose Estimation](https://mlsanity.com/paper/2003.04030/) (2020)
- [Data Efficient Reinforcement Learning for Legged Robots](https://mlsanity.com/paper/1907.03613/) (2019)
- [Language Instructed Reinforcement Learning for Human-AI Coordination](https://mlsanity.com/paper/2304.07297/) (2023)
- [Inferring Rewards from Language in Context](https://mlsanity.com/paper/2204.02515/) (2022)
- [Reinforcement Learning for UAV Attitude Control](https://mlsanity.com/paper/1804.04154/) (2018)
- [SayTap: Language to Quadrupedal Locomotion](https://mlsanity.com/paper/2306.07580/) (2023)
- [Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviors](https://mlsanity.com/paper/2203.17138/) (2022)
- [Learning Memory-Based Control for Human-Scale Bipedal Locomotion](https://mlsanity.com/paper/2006.02402/) (2020)
- [Imitation Learning from MPC for Quadrupedal Multi-Gait Control](https://mlsanity.com/paper/2103.14331/) (2021)
- [Cat-like Jumping and Landing of Legged Robots in Low-gravity Using Deep Reinforcement Learning](https://mlsanity.com/paper/2106.09357/) (2021)
- [Controlling the Solo12 Quadruped Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2309.16683/) (2023)
- [Reward-Adaptive Reinforcement Learning: Dynamic Policy Gradient Optimization for Bipedal Locomotion](https://mlsanity.com/paper/2107.01908/) (2021)
- [Learning predictive representations in autonomous driving to improve deep reinforcement learning](https://mlsanity.com/paper/2006.15110/) (2020)
- [Teach Biped Robots to Walk via Gait Principles and Reinforcement Learning with Adversarial Critics](https://mlsanity.com/paper/1910.10194/) (2019)

## Built on by (most cited first, 9 of 9)

- [PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learning](https://mlsanity.com/paper/2511.13765/) (2025)
- [Beyond Pixels: Learning Invariant Rewards for Real-World Robotics From a Few Demonstrations](https://mlsanity.com/paper/2605.22123/) (2026)
- [Embodied Learning of Reward for Musculoskeletal Control with Vision Language Models](https://mlsanity.com/paper/2512.23077/) (2025)
- [CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning](https://mlsanity.com/paper/2607.01721/) (2026)
- [The Rise of Verbal Reinforcement Learning](https://mlsanity.com/paper/2609.01597/) (2026)
- [Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models](https://mlsanity.com/paper/2609.05401/) (2026)
- [No Free Checker: A Survey of Verifiers for Robot Policies](https://mlsanity.com/paper/2609.09250/) (2026)
- [Track-and-Complete: Learning Humanoid Skills from a Single Failed Human Video](https://mlsanity.com/paper/2609.36924/) (2026)
- [Video2STL: Grounding VLM-Generated Temporal Specifications for Robot Learning](https://mlsanity.com/paper/2609.37519/) (2026)

## Abstract (arXiv)

Learning behavior in legged robots presents a significant challenge due to its inherent instability and complex constraints. Recent research has proposed the use of a large language model (LLM) to generate reward functions in reinforcement learning, thereby replacing the need for manually designed rewards by experts. However, this approach, which relies on textual descriptions to define learning objectives, fails to achieve controllable and precise behavior learning with clear directionality. In this paper, we introduce a new video2reward method, which directly generates reward functions from videos depicting the behaviors to be mimicked and learned. Specifically, we first process videos containing the target behaviors, converting the motion information of individuals in the videos into keypoint trajectories represented as coordinates through a video2text transforming module. These trajectories are then fed into an LLM to generate the reward function, which in turn is used to train the policy. To enhance the quality of the reward function, we develop a video-assisted iterative reward refinement scheme that visually assesses the learned behaviors and provides textual feedback to the LLM. This feedback guides the LLM to continually refine the reward function, ultimately facilitating more efficient behavior learning. Experimental results on tasks involving bipedal and quadrupedal robot motion control demonstrate that our method surpasses the performance of state-of-the-art LLM-based reward generation methods by over 37.6% in terms of human normalized score. More importantly, by switching video inputs, we find our method can rapidly learn diverse motion behaviors such as walking and running.