mlsanityData to 14 Oct 2026

VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning

arXiv 2610.02801 ↗First posted 2 Oct 2026cs.AI cs.LG cs.ROcited by 0

On the map: Deep reinforcement learning methods → RL with diffusion and flow models also fits: Deep RL algorithms and representations

Abstract (arXiv)

Model-based reinforcement learning (MBRL) achieves strong sample efficiency by planning within learned latent dynamics, yet its performance degrades substantially under unseen visual distractions such as background variations, lighting changes, or camera shifts. Unlike model-free RL, where encoder perturbations affect only single-step predictions, MBRL suffers from a two-level vulnerability: visual distractions first push encoder outputs out of distribution, and these errors then compound through recursive latent rollouts over the planning horizon. We propose visual generalization via latent-space consistency in model-based RL (VIGOR), a framework that enables zero-shot generalization to unseen visual distractions while retaining the sample efficiency of its MBRL backbone. VIGOR integrates three interdependent components: (i) asymmetric weak-to-strong augmentation, which pairs weak-only and weak-to-strong latent views within a single batch; (ii) dynamics-level consistency, which enforces augmentation-invariant transition predictions through direct latent regression; and (iii) encoder-level stabilization, which prevents encoder drift under the cross-augmentation supervision imposed by dynamics-level consistency. Evaluations on the DeepMind Control Suite (DMC) and Robosuite show that VIGOR outperforms state-of-the-art model-free and model-based baselines, surpassing the second-best baseline by 3.4% on DMC and 43.6% on Robosuite. Ablations further show that VIGOR's robustness is augmentation-agnostic: replacing the default augmentation with alternatives from distinct perturbation families preserves strong generalization, confirming that latent-space consistency, not the augmentation choice, drives robustness.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods. Builds on 50 of its 56 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

50 of 56 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Diffusion for World Modeling: Visual Details Matter in Atari20 May

Other references · 49

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  5. 2018World Modelscited by 1,513
  6. 2015Spatial Transformer Networkscited by 1,429
  7. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  8. 2023Mastering Diverse Domains through World Modelscited by 1,253
  9. 2020Mastering Atari with Discrete World Modelscited by 1,007
  10. 2018DeepMind Control Suitecited by 956
  11. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  12. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
37 more
  1. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  2. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  3. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  4. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  5. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
  6. 2019Model-Based Reinforcement Learning for Ataricited by 502
  7. 2020Reinforcement Learning with Augmented Datacited by 468
  8. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  9. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  10. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  11. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  12. 2021Mastering Atari Games with Limited Datacited by 237
  13. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  14. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 137
  15. 2025Training Agents Inside of Scalable World Modelscited by 136
  16. 2020Self-Supervised Policy Adaptation during Deploymentcited by 123
  17. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 105
  18. 2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 104
  19. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 100
  20. 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 68
  21. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 60
  22. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 41
  23. 2020Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learningcited by 35
  24. 2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 34
  25. 2023RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalizationcited by 28
  26. 2022Sample Efficient Deep Reinforcement Learning via Uncertainty Estimationcited by 27
  27. 2024Learning Latent Dynamic Robust Representations for World Modelscited by 21
  28. 2023Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policiescited by 20
  29. 2023RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictabilitycited by 18
  30. 2023MoVie: Visual Model-Based Policy Adaptation for View Generalizationcited by 16
  31. 2023Improving Generalization in Visual Reinforcement Learning via Conflict-aware Gradient Agreement Augmentationcited by 14
  32. 2024Focus On What Matters: Separated Models For Visual-Based RL Generalizationcited by 6
  33. 2024Make the Pertinent Salient: Task-Relevant Reconstruction for Visual Control with Distractionscited by 5
  34. 2023Learning Generalizable Agents via Saliency-Guided Features Decorrelationcited by 4
  35. 2024Zero-Shot Generalization of Vision-Based RL Without Data Augmentationcited by 4
  36. 2025Dream to Generalize: Zero-Shot Model-Based Reinforcement Learning for Unseen Visual Distractionscited by 4
  37. 2025Self-Consistent Model-based Adaptation for Visual Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Salience-Invariant Consistent Policy Learning for Generalization in Visual Reinforcement Learning
  2. 2025Pre-trained Visual Representations Generalize Where it Matters in Model-Based Reinforcement Learning
  3. 2024The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning
  4. 2026KAGE-Bench: Fast Known-Axis Visual Generalization Evaluation for Reinforcement Learning
  5. 2024Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularization
  6. 2025Synthetic Data is Sufficient for Zero-Shot Visual Generalization from Offline Data
  7. 2025Self-Predictive Dynamics for Generalization of Vision-based Reinforcement Learning
  8. 2025Good Actions Succeed, Bad Actions Generalize: A Case Study on Why RL Generalizes Better