mlsanityData to 7 Oct 2026

DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control

arXiv 2409.12192 ↗First posted 18 Sep 2024cs.RO cs.AI cs.CVcited by 43

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Imitation learning has proven to be a powerful tool for training complex visuomotor policies. However, current methods often require hundreds to thousands of expert demonstrations to handle high-dimensional visual observations. A key reason for this poor data efficiency is that visual representations are predominantly either pretrained on out-of-domain data or trained directly through a behavior cloning objective. In this work, we present DynaMo, a new in-domain, self-supervised method for learning visual representations. Given a set of expert demonstrations, we jointly learn a latent inverse dynamics model and a forward dynamics model over a sequence of image embeddings, predicting the next frame in latent space, without augmentations, contrastive sampling, or access to ground truth actions. Importantly, DynaMo does not require any out-of-domain data such as Internet datasets or cross-embodied datasets. On a suite of six simulated and real environments, we show that representations learned with DynaMo significantly improve downstream imitation learning performance over prior self-supervised learning objectives, and pretrained representations. Gains from using DynaMo hold across policy classes such as Behavior Transformer, Diffusion Policy, MLP, and nearest neighbors. Finally, we ablate over key components of DynaMo and measure its impact on downstream policy performance. Robot videos are best viewed at https://dynamo-ssl.github.io

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 75 of its 99 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

75 of 99 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware23 Apr
  2. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training30 Sep
  3. 2022Behavior Transformers: Cloning k modes with one stone22 Jun
  4. 2021The Surprising Effectiveness of Representation Learning for Visual Imitation2 Dec
  5. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videos12 Aug
  6. 2021Playful Interactions for Representation Learning19 Jul
  7. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning25 Oct

Other references · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  10. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  12. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
38 more
  1. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  3. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  7. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  8. 2020Mastering Atari with Discrete World Modelscited by 990
  9. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  10. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  11. 2016Pixel Recurrent Neural Networkscited by 814
  12. 2024Genie: Generative Interactive Environmentscited by 706
  13. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  14. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  15. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  16. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  17. 2020Reinforcement Learning with Augmented Datacited by 452
  18. 2021Implicit Behavioral Cloningcited by 442
  19. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  20. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  21. 2022Masked Visual Pre-training for Motor Controlcited by 246
  22. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  23. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
  24. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  25. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  26. 2024Behavior Generation with Latent Actionscited by 159
  27. 2022Human-to-Robot Imitation in the Wildcited by 157
  28. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  29. 2019Temporal Cycle-Consistency Learningcited by 151
  30. 2023Denoising Diffusion Autoencoders are Unified Self-supervised Learnerscited by 128
  31. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  32. 2023Learning to Act without Actionscited by 116
  33. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  34. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  35. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  36. 2018Imitating Latent Policies from Observationcited by 101
  37. 2023On Bringing Robots Homecited by 101
  38. 2024BAKU: An Efficient Transformer for Multi-Task Policy Learningcited by 80

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 23; 2026: 19.

43 papers.

Sort:

By area:

  1. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  2. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  3. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 65
  4. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 58
  5. 2026Learning Latent Action World Models In The Wildcited by 42
  6. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 40
  7. 2025CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrationscited by 39
  8. 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 37
  9. 20253D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Taskscited by 30
  10. 2025What Do Latent Action Models Actually Learn?cited by 30
  11. 2025Latent Diffusion Planning for Imitation Learningcited by 26
  12. 2025ViPRA: Video Prediction for Robot Actionscited by 24
31 more
  1. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  2. 2026Sensorimotor World Models: Perception for Action via Inverse Dynamicscited by 14
  3. 2025PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planningcited by 13
  4. 2025TraceGen: World Modeling in 3D Trace Space Enables Learning from Cross-Embodiment Videoscited by 13
  5. 2025Learning Interactive World Model for Object-Centric Reinforcement Learningcited by 11
  6. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 11
  7. 2025Object-Centric Latent Action Learningcited by 9
  8. 2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 9
  9. 2026DeltaVLA: Prior-Guided Vision-Language-Action Models via World Knowledge Variationcited by 9
  10. 2026What Makes Video World Model Latents Action-Relevant: Prediction over Reconstructioncited by 8
  11. 2025ACT-JEPA: Novel Joint-Embedding Predictive Architecture for Efficient Policy Representation Learningcited by 7
  12. 2025LatBot: Distilling Universal Latent Actions for Vision-Language-Action Modelscited by 7
  13. 2025Robotic VLA Benefits from Joint Learning with Motion Image Diffusioncited by 7
  14. 2025Latent Action World Models for Control with Unlabeled Trajectoriescited by 5
  15. 2026UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Modelscited by 3
  16. 2026ALAM: Algebraically Consistent Latent Action Model for Vision-Language-Action Modelscited by 3
  17. 2025Midway Network: Learning Representations for Recognition and Motion from Latent Dynamicscited by 2
  18. 2026From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulationcited by 2
  19. 2026What Matters for Latent Actions in Robot Learningcited by 2
  20. 2025Seeing Space and Motion: Enhancing Latent Actions with Geometric and Dynamic Awareness for Vision-Language-Action Modelscited by 1
  21. 2026Vision-Language Models Unlock Task-Centric Latent Actionscited by 1
  22. 2026Why Latent Actions Fail, and How to Prevent Itcited by 1
  23. 2026Learning Action Priors for Cross-embodiment Robot Manipulationcited by 1
  24. 2025Object-Centric Action-Enhanced Representations for Robot Visuo-Motor Policy Learningcited by 0
  25. 2026LA-Pose: Latent Action Pretraining Meets Pose Estimationcited by 0
  26. 2026Mitigating State Aliasing in Vision-Language-Action Models via Inverse Dynamics Learningcited by 0
  27. 2026Output-Level Regularization Eliminates the Seed Lottery in Single-GPU VLA Fine-Tuningcited by 0
  28. 2026Patch Policy: Efficient Embodied Control via Dense Visual Representationscited by 0
  29. 2026Latent-Action-Guided Video-Language Feature Learning for Surgical Instrument-Tissue Interaction Recognitioncited by 0
  30. 2026Dueling World Models: Advantage-Style Action Channels for Common-Mode Distractor Rejectioncited by 0
  31. 2026Proprioception-Anchored Cross-Modal Pretraining for Zero-Shot Sim-to-Real Contact-Rich Assemblycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019Self-Supervised Correspondence in Visuomotor Policy Learning
  2. 2025Correspondence-Oriented Imitation Learning: Flexible Visuomotor Control with 3D Conditioning
  3. 2025DINOv3-Diffusion Policy: Self-Supervised Large Visual Model for Visuomotor Diffusion Policy Learning
  4. 2026VPWEM: Non-Markovian Visuomotor Policy with Working and Episodic Memory
  5. 2026Latent Policy Steering: An Efficient and Flexible Framework for Cross-Embodiment Transfer ↗
  6. 2025Latent Policy Barrier: Learning Robust Visuomotor Policies by Staying In-Distribution
  7. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skills
  8. 2023SpawnNet: Learning Generalizable Visuomotor Skills from Pre-trained Networks