mlsanityData to 7 Oct 2026

Multi-view Disentanglement for Reinforcement Learning with Multiple Cameras

arXiv 2404.14064 ↗First posted 22 Apr 2024cs.LG cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

The performance of image-based Reinforcement Learning (RL) agents can vary depending on the position of the camera used to capture the images. Training on multiple cameras simultaneously, including a first-person egocentric camera, can leverage information from different camera perspectives to improve the performance of RL. However, hardware constraints may limit the availability of multiple cameras in real-world deployment. Additionally, cameras may become damaged in the real-world preventing access to all cameras that were used during training. To overcome these hardware constraints, we propose Multi-View Disentanglement (MVD), which uses multiple cameras to learn a policy that is robust to a reduction in the number of cameras to generalise to any single camera from the training set. Our approach is a self-supervised auxiliary task for RL that learns a disentangled representation from multiple cameras, with a shared representation that is aligned across all cameras to allow generalisation to a single camera, and a private representation that is camera-specific. We show experimentally that an RL agent trained on a single third-person camera is unable to learn an optimal policy in many control tasks; but, our approach, benefiting from multiple cameras during training, is able to solve the task using only the same single third-person camera.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 40 of its 53 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

40 of 53 references.

References · 40

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  6. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  8. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  9. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  10. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  11. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  12. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 542
28 more
  1. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  2. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  3. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 147
  4. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  5. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  6. 2018Learning to Decompose and Disentangle Representations for Video Predictioncited by 120
  7. 2016A Survey of Multi-View Representation Learningcited by 107
  8. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 107
  9. 2020Deep Reinforcement and InfoMax Learningcited by 79
  10. 2022Impossibility Theorems for Feature Attributioncited by 78
  11. 2018Image-to-image translation for cross-domain disentanglementcited by 61
  12. 2020Predictive Information Accelerates Learning in RLcited by 56
  13. 2021panda-gym: Open-source goal-conditioned environments for robotic learningcited by 56
  14. 2021Multi-VAE: Learning Disentangled View-common and View-peculiar Visual Representations for Multi-view Clusteringcited by 51
  15. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  16. 2022Vision-Based Manipulators Need to Also See from Their Handscited by 46
  17. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  18. 2023Conditional Mutual Information for Disentangled Representations in Reinforcement Learningcited by 27
  19. 2022Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?cited by 24
  20. 2021Self-Supervised Disentangled Representation Learning for Third-Person Imitation Learningcited by 23
  21. 2022Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learningcited by 19
  22. 2024DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Designcited by 11
  23. 2022Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learningcited by 7
  24. 2023Visual-Policy Learning through Multi-Camera View to Single-Camera View Knowledge Distillation for Robot Manipulation Taskscited by 7
  25. 2019An Actor-Critic-Attention Mechanism for Deep Reinforcement Learning in Multi-view Environmentscited by 5
  26. 2023Disentangling Multi-view Representations Beyond Inductive Biascited by 4
  27. 2020Interpretable Deep Representation Learning from Temporal Multi-view Datacited by 2
  28. 2023Self-supervised Multi-view Disentanglement for Expansion of Visual Collectionscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 1.

8 papers.

  1. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  2. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8
  3. 2025ManiVID-3D: Generalizable View-Invariant Reinforcement Learning for Robotic Manipulation via Disentangled 3D Representationscited by 5
  4. 2025Enhancing Tactile-based Reinforcement Learning for Robotic Controlcited by 5
  5. 2025Merging and Disentangling Views in Visual Reinforcement Learning for Robotic Manipulationcited by 2
  6. 2025Learning Fused State Representations for Control from Multi-View Observationscited by 1
  7. 2026Reinforcement Learning-Guided Retrieval with Soft Fusion for Robust Multimodal Imitation Learning under Missing Modalitiescited by 1
  8. 2024Skill-aware Mutual Information Optimisation for Generalisation in Reinforcement Learningcited by 0