mlsanityData to 7 Oct 2026

VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training

arXiv 2210.00030 ↗First posted 30 Sep 2022cs.RO cs.AI cs.CVcited by 384

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Reward and representation learning are two long-standing challenges for learning an expanding set of robot manipulation skills from sensory observations. Given the inherent cost and scarcity of in-domain, task-specific robot data, learning from large, diverse, offline human videos has emerged as a promising path towards acquiring a generally useful visual representation for control; however, how these human videos can be used for general-purpose reward learning remains an open question. We introduce Value-Implicit Pre-training (VIP), a self-supervised pre-trained visual representation capable of generating dense and smooth reward functions for unseen robotic tasks. VIP casts representation learning from human videos as an offline goal-conditioned reinforcement learning problem and derives a self-supervised dual goal-conditioned value-function objective that does not depend on actions, enabling pre-training on unlabeled human videos. Theoretically, VIP can be understood as a novel implicit time contrastive objective that generates a temporally smooth embedding, enabling the value function to be implicitly defined via the embedding distance, which can then be used to construct the reward for any goal-image specified downstream task. Trained on large-scale Ego4D human videos and without any fine-tuning on in-domain, task-specific data, VIP's frozen representation can provide dense visual reward for an extensive set of simulated and real-robot tasks, enabling diverse reward-based visual control methods and significantly outperforming all prior pre-trained representations. Notably, VIP can enable simple, few-shot offline RL on a suite of real-world robot tasks with as few as 20 trajectories.

Key facts. Cited by 384 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 49 of its 71 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

49 of 71 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb
  2. 2021XIRL: Cross-embodiment Inverse Reinforcement Learning7 Jun
  3. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning25 Oct
  4. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation7 Nov
  5. 2018Zero-Shot Visual Imitation23 Apr

Other references · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  12. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
32 more
  1. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  2. 2017Hindsight Experience Replaycited by 1,096
  3. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  5. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  6. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  7. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  8. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  9. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  10. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  11. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  12. 2022Masked Visual Pre-training for Motor Controlcited by 246
  13. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  14. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  15. 2022Human-to-Robot Imitation in the Wildcited by 157
  16. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  17. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  18. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  19. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  20. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  21. 2021RRL: Resnet as representation for Reinforcement Learningcited by 89
  22. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  23. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  24. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 80
  25. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  26. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 68
  27. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  28. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 64
  29. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
  30. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  31. 2022Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learningcited by 16
  32. 2021PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulatorscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 75; 2024: 109; 2025: 90; 2026: 107.

384 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  3. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  4. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  5. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  6. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  7. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  8. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  9. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  10. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  11. 2023A Cookbook of Self-Supervised Learningcited by 248
  12. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
38 more
  1. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  2. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  3. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  4. 2024RT-H: Action Hierarchies Using Languagecited by 204
  5. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 191
  6. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  7. 2023Language-Driven Representation Learning for Roboticscited by 163
  8. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  9. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 160
  10. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulationcited by 159
  11. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  12. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  13. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 144
  14. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  15. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  16. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  17. 2023Vision-Language Models as Success Detectorscited by 138
  18. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  19. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 132
  20. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  21. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  22. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  23. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  24. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  25. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  26. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  27. 2023On Bringing Robots Homecited by 101
  28. 2024Foundation Policies with Hilbert Representationscited by 89
  29. 2023An Outlook into the Future of Egocentric Visioncited by 84
  30. 2024Vision Language Models are In-Context Value Learnerscited by 84
  31. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  32. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  33. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 78
  34. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  35. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  36. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 66
  37. 2024In-Context Imitation Learning via Next-Token Predictioncited by 65
  38. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 65

50 most cited of 384 All 384 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ViPRA: Video Prediction for Robot Actions
  2. 2024Rank2Reward: Learning Shaped Reward Functions from Passive Video
  3. 2024Representation Alignment from Human Feedback for Cross-Embodiment Reward Learning from Mixed-Quality Demonstrations
  4. 2024Learning Manipulation Skills through Robot Chain-of-Thought with Sparse Failure Guidance
  5. 2025MA-ROESL: Motion-aware Rapid Reward Optimization for Efficient Robot Skill Learning from Single Videos ↗
  6. 2021The Surprising Effectiveness of Representation Learning for Visual Imitation
  7. 2026Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
  8. 2022Reinforcement Learning with Action-Free Pre-Training from Videos