mlsanityData to 14 Oct 2026

IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AI

arXiv 2411.00785 ↗First posted 17 Oct 2024cs.RO cs.AIcited by 72

On the map: World models → Embodied and robot world models

Abstract (arXiv)

We introduce Image-GOal Representations (IGOR), aiming to learn a unified, semantically consistent action space across human and various robots. Through this unified latent action space, IGOR enables knowledge transfer among large-scale robot and human activity data. We achieve this by compressing visual changes between an initial image and its goal state into latent actions. IGOR allows us to generate latent action labels for internet-scale video data. This unified latent action space enables the training of foundation policy and world models across a wide variety of tasks performed by both robots and humans. We demonstrate that: (1) IGOR learns a semantically consistent action space for both human and robots, characterizing various possible motions of objects representing the physical interaction knowledge; (2) IGOR can "migrate" the movements of the object in the one video to other videos, even across human and robots, by jointly using the latent action model and world model; (3) IGOR can learn to align latent actions with natural language through the foundation policy model, and integrate latent actions with a low-level policy model to achieve effective robot control. We believe IGOR opens new possibilities for human-to-robot knowledge transfer and control.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 48 of its 51 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

48 of 51 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Models24 May
  2. 2023Learning to Act without Actions17 Dec
  3. 2023Structured World Models from Human Videos21 Aug
  4. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  10. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  11. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  12. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
32 more
  1. 2024Genie: Generative Interactive Environmentscited by 747
  2. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  3. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  4. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  5. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  6. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  7. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  8. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  9. 2022Masked Visual Pre-training for Motor Controlcited by 248
  10. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  11. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 225
  12. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  13. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  14. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 204
  15. 2020Spatial-Temporal Transformer Networks for Traffic Flow Forecastingcited by 180
  16. 2023Language-Driven Representation Learning for Roboticscited by 165
  17. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 164
  18. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 133
  19. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  20. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 108
  21. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  22. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  23. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 100
  24. 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokenscited by 89
  25. 2021Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulationcited by 83
  26. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  27. 2024FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learningcited by 80
  28. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learningcited by 77
  29. 2023MUTEX: Learning Unified Policies from Multimodal Task Specificationscited by 70
  30. 2023Multi-Stage Cable Routing through Hierarchical Imitation Learningcited by 60
  31. 2023HYDRA: Hybrid Robot Actions for Imitation Learningcited by 55
  32. 2024FlowRetrieval: Flow-Guided Data Retrieval for Few-Shot Imitation Learningcited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 28; 2026: 39.

72 papers.

Sort:

By area:

  1. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  2. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 289
  3. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulationcited by 161
  4. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
  5. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 118
  6. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  7. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 108
  8. 2025UP-VLA: A Unified Understanding and Prediction Model for Embodied Agentcited by 102
  9. 2025VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizerscited by 70
  10. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 70
  11. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 69
  12. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 68
38 more
  1. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 63
  2. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 44
  3. 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 40
  4. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 36
  5. 2025WholeBodyVLA: Towards Unified Latent VLA for Whole-Body Loco-Manipulation Controlcited by 36
  6. 2024VidTok: A Versatile and Open-Source Video Tokenizercited by 35
  7. 2025What Do Latent Action Models Actually Learn?cited by 32
  8. 2025PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generationcited by 23
  9. 2026A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulationcited by 22
  10. 2025Playing with Transformer at 30+ FPS via Next-Frame Diffusioncited by 20
  11. 2025Is Diversity All You Need for Scalable Robotic Manipulation?cited by 20
  12. 2026Joint-Aligned Latent Action: Towards Scalable VLA Pretraining in the Wildcited by 15
  13. 2025STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantizationcited by 13
  14. 2026VideoWorld 2: Learning Transferable Knowledge from Real-world Videoscited by 12
  15. 2026LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignmentcited by 12
  16. 2025GAF: Gaussian Action Field as a 4D Representation for Dynamic World Modeling in Robotic Manipulationcited by 11
  17. 2026ProgressVLA: Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulationcited by 10
  18. 2026DiLA: Disentangled Latent Action World Modelscited by 10
  19. 2025Object-Centric Latent Action Learningcited by 9
  20. 2025Fast Autoregressive Video Generation with Diagonal Decodingcited by 8
  21. 2026RotVLA: Rotational Latent Action for Vision-Language-Action Modelcited by 8
  22. 2026Learning Additively Compositional Latent Actions for Embodied AIcited by 7
  23. 2025How Do VLAs Effectively Inherit from VLMs?cited by 6
  24. 2025See Once, Then Act: Vision-Language-Action Model with Task Learning from One-Shot Video Demonstrationscited by 6
  25. 2025Incentivizing Multimodal Reasoning in Large Models for Direct Robot Manipulationcited by 5
  26. 2025Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representationscited by 5
  27. 2026UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Modelscited by 5
  28. 2026Robot Learning from Human Videos: A Surveycited by 5
  29. 2026UAM: A Dual-Stream Perspective on Forgetting in VLA Trainingcited by 5
  30. 2026HARP-VLA: Human-Robot Aligned Representation Learning for Vision-Language-Action Modelcited by 5
  31. 2026GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulationcited by 5
  32. 2025Hyper-GoalNet: Goal-Conditioned Manipulation Policy Learning with HyperNetworkscited by 4
  33. 2026From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Modelscited by 4
  34. 2026What Matters for Latent Actions in Robot Learningcited by 4
  35. 2025Beyond Reward Margin: Rethinking and Resolving Likelihood Displacement in Diffusion Models via Video Generationcited by 3
  36. 2026Vision-Language Models Unlock Task-Centric Latent Actionscited by 3
  37. 2026ALAM: Algebraically Consistent Latent Action Model for Vision-Language-Action Modelscited by 3
  38. 2026From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Datacited by 3

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Being-H0.7: A Latent World-Action Model from Egocentric Videos
  2. 2026Native Video-Action Pretraining for Generalizable Robot Control
  3. 2026Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models
  4. 2025UniJEPA: Enhancing Robot Policy via Unified Continuous and Discrete Representation Learning
  5. 2025LatBot: Distilling Universal Latent Actions for Vision-Language-Action Models
  6. 2026InternW0-Δ: A World Action Model Bridging Predictive Dynamics and Actions with 20K+ Hours of Open Data
  7. 2026WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos
  8. 2026LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies