mlsanityData to 7 Oct 2026

Object-Centric Latent Action Learning

arXiv 2502.09680 ↗First posted 13 Feb 2025cs.CV cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Leveraging vast amounts of unlabeled internet video data for embodied AI is currently bottlenecked by the lack of action labels and the presence of action-correlated visual distractors. Although recent latent action policy optimization (LAPO) has shown promise in inferring proxy action labels from visual observations, its performance degrades significantly when distractors are present. To address this limitation, we propose a novel object-centric latent action learning framework that centers on objects rather than pixels. We leverage self-supervised object-centric pretraining to disentangle the movement of the agent and distracting background dynamics. This allows LAPO to focus on task-relevant interactions, resulting in more robust proxy-action labels, enabling better imitation learning and efficient adaptation of the agent with just a few action-labeled trajectories. We evaluated our method in eight visually complex tasks across the Distracting Control Suite (DCS) and Distracting MetaWorld (DMW). Our results show that object-centric pretraining mitigates the negative effects of distractors by 50%, as measured by downstream task performance: average return (DCS) and success rate (DMW).

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 33 of its 43 references that are in the map. First posted on arXiv 13 Feb 2025.

Builds on

33 of 43 references.

References · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  8. 2020Object-Centric Learning with Slot Attentioncited by 865
  9. 2024Genie: Generative Interactive Environmentscited by 706
  10. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  11. 2024Latent Action Pretraining from Videoscited by 302
  12. 2021Conditional Object-Centric Learning from Videocited by 247
21 more
  1. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 144
  2. 2022Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videoscited by 139
  3. 2023Learning to Act without Actionscited by 116
  4. 2018Imitating Latent Policies from Observationcited by 101
  5. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
  6. 2021Generalization and Robustness Implications in Object-Centric Learningcited by 81
  7. 2023Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similaritiescited by 78
  8. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  9. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 64
  10. 2024Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imaginationcited by 45
  11. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 43
  12. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  13. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 40
  14. 2024Generative Omnimatte: Learning to Decompose Video into Layerscited by 38
  15. 2024Adaptive Slot Attention: Object Discovery with Dynamic Slot Numbercited by 34
  16. 2021Provable RL with Exogenous Distractors via Multistep Inverse Dynamicscited by 28
  17. 2024Benchmarking Vision, Language, & Action Models on Robotic Learning Taskscited by 23
  18. 2025Slot-Guided Adaptation of Pre-trained Diffusion Models for Object-Centric Learning and Compositional Generationcited by 13
  19. 2025PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planningcited by 13
  20. 2024AD3: Implicit Action is the Key for World Models to Distinguish the Diverse Visual Distractorscited by 10
  21. 2024Towards Principled Representation Learning from Videos for Reinforcement Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 8.

9 papers.

  1. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 104
  2. 2025CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrationscited by 39
  3. 2026MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstructioncited by 11
  4. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 11
  5. 2026Factored Latent Action World Modelscited by 6
  6. 2026Segment to Focus: Guiding Latent Action Models in the Presence of Distractorscited by 5
  7. 2026Vision-Language Models Unlock Task-Centric Latent Actionscited by 1
  8. 2026KAGE-Bench: Fast Known-Axis Visual Generalization Evaluation for Reinforcement Learningcited by 0
  9. 2026Latent Actions from Factorized Transition Effects under Agent Ambiguitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Object-Centric Action-Enhanced Representations for Robot Visuo-Motor Policy Learning
  2. 2025LAOF: Robust Latent Action Learning with Optical Flow Constraints
  3. 2025Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations
  4. 2026LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignment
  5. 2026Causally Debiased Latent Action Model for Embodied Action Conditioned World Models
  6. 2026LAFP: Preserving Latent Action Structure in Latent Policy Learning via Flow Matching
  7. 2025Latent Action Pretraining Through World Modeling
  8. 2026Learning Latent Action World Models In The Wild