mlsanityData to 7 Oct 2026

AD3: Implicit Action is the Key for World Models to Distinguish the Diverse Visual Distractors

arXiv 2403.09976 ↗First posted 15 Mar 2024cs.LG cs.CVcited by 10

On the map: World models → Embodied and robot world models

Abstract (arXiv)

Model-based methods have significantly contributed to distinguishing task-irrelevant distractors for visual control. However, prior research has primarily focused on heterogeneous distractors like noisy background videos, leaving homogeneous distractors that closely resemble controllable agents largely unexplored, which poses significant challenges to existing methods. To tackle this problem, we propose Implicit Action Generator (IAG) to learn the implicit actions of visual distractors, and present a new algorithm named implicit Action-informed Diverse visual Distractors Distinguisher (AD3), that leverages the action inferred by IAG to train separated world models. Implicit actions effectively capture the behavior of background distractors, aiding in distinguishing the task-irrelevant components, and the agent can optimize the policy within the task-relevant state space. Our method achieves superior performance on various visual control tasks featuring both heterogeneous and homogeneous distractors. The indispensable role of implicit actions learned by IAG is also empirically validated.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 41 of its 48 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

41 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Learning to Act without Actions17 Dec
  2. 2023Structured World Models from Human Videos21 Aug

Other references · 39

  1. 2023Segment Anythingcited by 8,158
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  4. 2017The Kinetics Human Action Video Datasetcited by 1,894
  5. 2016OpenAI Gymcited by 1,684
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2018World Modelscited by 1,444
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  9. 2023Mastering Diverse Domains through World Modelscited by 1,193
  10. 2020Mastering Atari with Discrete World Modelscited by 990
  11. 2018DeepMind Control Suitecited by 935
  12. 2016Deep Variational Information Bottleneckcited by 915
27 more
  1. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  2. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  3. 2024Genie: Generative Interactive Environmentscited by 706
  4. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  5. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  6. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  7. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  8. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  9. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  10. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  11. 2019Temporal Cycle-Consistency Learningcited by 151
  12. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  13. 2018Imitating Latent Policies from Observationcited by 101
  14. 2022Deep Hierarchical Planning from Pixelscited by 96
  15. 2019Learning Action Representations for Reinforcement Learningcited by 75
  16. 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 66
  17. 2022A Survey on Model-based Reinforcement Learningcited by 65
  18. 2021Learning Task Informed Abstractionscited by 58
  19. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  20. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Modelscited by 53
  21. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  22. 2023Disentanglement via Latent Quantizationcited by 40
  23. 2021LASER: Learning a Latent Action Space for Efficient Reinforcement Learningcited by 32
  24. 2021Action-Sufficient State Representation Learning for Control with Structural Constraintscited by 31
  25. 2021Provable RL with Exogenous Distractors via Multistep Inverse Dynamicscited by 28
  26. 2022INFOrmation Prioritization through EmPOWERment in Visual Model-Based RLcited by 27
  27. 2023SeMAIL: Eliminating Distractors in Visual Imitation via Separated Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 40
  2. 2025What Do Latent Action Models Actually Learn?cited by 30
  3. 2025Co-Evolving Latent Action World Modelscited by 12
  4. 2025DMWM: Dual-Mind World Model with Long-Term Imaginationcited by 10
  5. 2025Object-Centric Latent Action Learningcited by 9
  6. 2025Reimagination with Test-time Observation Interventions: Distractor-Robust World Model Predictions for Visual Model Predictive Controlcited by 6
  7. 2026Segment to Focus: Guiding Latent Action Models in the Presence of Distractorscited by 5
  8. 2026Temporally Centered SIGReg Improves LeWorldModel Representations for Robot Policy Learningcited by 4
  9. 2026Diagnosing JEPA World Models with Action-Conditioned Predictive Consistencycited by 2
  10. 2026Dueling World Models: Advantage-Style Action Channels for Common-Mode Distractor Rejectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies
  2. 2025Action Anticipation at a Glimpse: To What Extent Can Multimodal Cues Replace Video?
  3. 2026CtrlWAM: Controllable World Action Models with Aligned Intent and Foresight
  4. 2026Making Foresight Actionable: Repurposing Representation Alignment in World Action Models
  5. 2026Vision-Language Models Unlock Task-Centric Latent Actions
  6. 2026TaskSense: Focusing on What Matters in World Models
  7. 2026WorldExam: Benchmarking World Models from Apparent Appearance to Inherent Reactivity
  8. 2026Olaf-World: Orienting Latent Actions for Video World Modeling