mlsanityData to 14 Oct 2026

Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation

arXiv 2312.13139 ↗First posted 20 Dec 2023cs.RO cs.CVcited by 364

Not on the map. Why?

Cited by 152 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generative pre-trained models have demonstrated remarkable effectiveness in language and vision domains by learning useful representations. In this paper, we extend the scope of this effectiveness by showing that visual robot manipulation can significantly benefit from large-scale video generative pre-training. We introduce GR-1, a straightforward GPT-style model designed for multi-task language-conditioned visual robot manipulation. GR-1 takes as inputs a language instruction, a sequence of observation images, and a sequence of robot states. It predicts robot actions as well as future images in an end-to-end manner. Thanks to a flexible design, GR-1 can be seamlessly finetuned on robot data after pre-trained on a large-scale video dataset. We perform extensive experiments on the challenging CALVIN benchmark and a real robot. On CALVIN benchmark, our method outperforms state-of-the-art baseline methods and improves the success rate from 88.9% to 94.9%. In the setting of zero-shot unseen scene generalization, GR-1 improves the success rate from 53.3% to 85.4%. In real robot experiments, GR-1 also outperforms baseline methods and shows strong potentials in generalization to unseen scenes and objects. We provide inaugural evidence that a unified GPT-style transformer, augmented with large-scale video generative pre-training, exhibits remarkable generalization to multi-task visual robot manipulation. Project page: https://GR1-Manipulation.github.io

Key facts. Cited by 364 papers in the map (data through 14 Oct 2026). Builds on 57 of its 62 references that are in the map. First posted on arXiv 20 Dec 2023.

Builds on

57 of 62 references.

References · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  12. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
38 more
  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  4. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  5. 2020Mastering Atari with Discrete World Modelscited by 1,007
  6. 2022A Generalist Agentcited by 808
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  8. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  9. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  10. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  11. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  12. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  13. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  14. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  15. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  16. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  17. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  18. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  19. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  20. 2022Behavior Transformers: Cloning k modes with one stonecited by 304
  21. 2016Deep Visual Foresight for Planning Robot Motioncited by 285
  22. 2022Masked Visual Pre-training for Motor Controlcited by 248
  23. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  24. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  25. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
  26. 2022Masked World Models for Visual Controlcited by 170
  27. 2023Language-Driven Representation Learning for Roboticscited by 165
  28. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 159
  29. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  30. 2023Structured World Models from Human Videoscited by 147
  31. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 138
  32. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 114
  33. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 111
  34. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  35. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  36. 2022Instruction-driven history-aware policies for robotic manipulationscited by 94
  37. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  38. 2023Video Prediction Models as Rewards for Reinforcement Learningcited by 78

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 51; 2025: 128; 2026: 185.

364 papers.

Sort:

By area:

  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  2. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  3. 2026World Action Models are Zero-shot Policiescited by 465
  4. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
  5. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 433
  6. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 422
  7. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  8. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 367
  9. 2026Fast-WAM: Do World Action Models Need Test-time Future Imagination?cited by 349
  10. 2024Latent Action Pretraining from Videoscited by 321
  11. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 303
  12. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 296
38 more
  1. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 289
  2. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 282
  3. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  4. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 240
  5. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  6. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
  7. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 181
  8. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 179
  9. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  10. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 165
  11. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulationcited by 161
  12. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 153
  13. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 138
  14. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 138
  15. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 134
  16. 2025Unified Vision-Language-Action Modelcited by 134
  17. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 120
  18. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 117
  19. 2025FLARE: Robot Learning with Implicit World Modelingcited by 113
  20. 2025GR-3 Technical Reportcited by 111
  21. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 108
  22. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 103
  23. 2025UP-VLA: A Unified Understanding and Prediction Model for Embodied Agentcited by 102
  24. 2024What Matters in Building Vision-Language-Action Models for Generalist Robotscited by 89
  25. 2024LLARVA: Vision-Action Instruction Tuning Enhances Robot Learningcited by 87
  26. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  27. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 82
  28. 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systemscited by 81
  29. 2025ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceivercited by 81
  30. 2025BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Modelscited by 80
  31. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 75
  32. 2026Being-H0.7: A Latent World-Action Model from Egocentric Videoscited by 75
  33. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 74
  34. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 74
  35. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 73
  36. 2026Unified 4D World Action Modeling from Video Priors with Asynchronous Denoisingcited by 71
  37. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 70
  38. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 70

50 most cited of 364 All 364 (JSON)