Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videos

arXiv 2410.09286 ↗First posted 11 Oct 2024cs.RO cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

Learning from Demonstrations, particularly from biological experts like humans and animals, often encounters significant data acquisition challenges. While recent approaches leverage internet videos for learning, they require complex, task-specific pipelines to extract and retarget motion data for the agent. In this work, we introduce a language-model-assisted bi-level programming framework that enables a reinforcement learning agent to directly learn its reward from internet videos, bypassing dedicated data preparation. The framework includes two levels: an upper level where a vision-language model (VLM) provides feedback by comparing the learner's behavior with expert videos, and a lower level where a large language model (LLM) translates this feedback into reward updates. The VLM and LLM collaborate within this bi-level framework, using a "chain rule" approach to derive a valid search direction for reward learning. We validate the method for reward learning from YouTube videos, and the results have shown that the proposed method enables efficient reward design from expert videos of biological agents for complex behavior synthesis.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 46 of its 64 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

46 of 64 references.

References · 46

2017Proximal Policy Optimization Algorithmscited by 14,441
2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396

34 more

2023Language to Rewards for Robotic Skill Synthesiscited by 307
2023Reward Design with Language Modelscited by 257
2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 229
2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
2018Markerless tracking of user-defined features with deep learningcited by 203
2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
2022Human-to-Robot Imitation in the Wildcited by 157
2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 132
2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradientscited by 122
2022Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtubecited by 79
2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 78
2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 66
2023Vision-Language Models as a Source of Rewardscited by 65
2016Unsupervised Perceptual Rewards for Imitation Learningcited by 64
2020Model-Based Inverse Reinforcement Learning from Visual Demonstrationscited by 52
2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 45
2023Creative Robot Tool Use with Large Language Modelscited by 42
2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
2024RL-GPT: Integrating Reinforcement Learning and Code-as-policycited by 32
2024ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Datacited by 29
2022Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviorscited by 27
2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 23
2024ExploRLLM: Guiding Exploration in Reinforcement Learning with Large Language Modelscited by 19
2024How Can LLM Guide RL? A Value-Based Approachcited by 13
2022Receding Horizon Inverse Reinforcement Learningcited by 10
2023SLoMo: A General System for Legged Robot Motion Imitation from Casual Videoscited by 10
2020Learning from Sparse Demonstrationscited by 9
2023Learning Reward for Physical Skills using Large Language Modelcited by 5
2024Visual Grounding for Object-Level Generalization in Reinforcement Learningcited by 4
2022Imitation and Adaptation Based on Consistency: A Quadruped Robot Imitates Animals from Videos Using Deep Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

Similar, not cited

Close in wording, no citation either way