mlsanityData to 14 Oct 2026

Multimodal Fusion for Sim2real Transfer in Visual Reinforcement Learning

arXiv 2507.09180 ↗First posted 12 Jul 2025cs.CV cs.ROcited by 0

Not on the map. Why?

Abstract (arXiv)

Depth information is robust to scene appearance variations and inherently carries 3D spatial details. Thus, a visual backbone based on the vision transformer is proposed to fuse RGB and depth modalities for enhancing generalization in this paper. Different modalities are first processed by separate CNN stems, and the combined convolutional features are delivered to the scalable vision transformer to obtain visual representations. Moreover, a contrastive learning scheme is designed with masked and unmasked tokens to enhance the sample efficiency and generalization performance. A curriculum-based domain randomization scheme is used to flexibly stabilize the training process. Finally, simulation results demonstrate that our fusion scheme outperforms the other baselines. The feasibility of our model is validated to perform real-world manipulation tasks via zero-shot transfer.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 44 references that are in the map. First posted on arXiv 12 Jul 2025.

Builds on

34 of 44 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  6. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  8. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  10. 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 582
  11. 2020Reinforcement Learning with Augmented Datacited by 468
  12. 2021Early Convolutions Help Transformers See Bettercited by 396
22 more
  1. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  2. 2022Masked Visual Pre-training for Motor Controlcited by 248
  3. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  4. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 231
  5. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  6. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 105
  7. 2019Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Taskscited by 67
  8. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 60
  9. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  10. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 47
  11. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  12. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 41
  13. 2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 34
  14. 2023RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalizationcited by 28
  15. 2023The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learningcited by 26
  16. 2024Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasetscited by 23
  17. 2022Analysis of Randomization Effects on Sim2Real Transfer in Reinforcement Learning for Robotic Manipulation Taskscited by 14
  18. 2022Spatio-channel Attention Blocks for Cross-modal Crowd Countingcited by 12
  19. 2023Efficient RL via Disentangled Environment and Agent Representationscited by 12
  20. 2024DEAR: Disentangled Environment and Agent Representations for Reinforcement Learning without Reconstructioncited by 6
  21. 2022Sim-to-Real Transfer of Robotic Assembly with Visual Inputs Using CycleGAN and Force Controlcited by 5
  22. 2025Grasping Deformable Objects via Reinforcement Learning with Cross-Modal Attention to Visuo-Tactile Inputscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Zero-Shot Visual Generalization in Robot Manipulation
  2. 2025DeGuV: Depth-Guided Visual Reinforcement Learning for Generalization and Interpretability in Manipulation
  3. 2026Pixel2Catch: Multi-Agent Sim-to-Real Transfer for Agile Manipulation with a Single RGB Camera ↗
  4. 2024Sim-To-Real Transfer for Visual Reinforcement Learning of Deformable Object Manipulation for Robot-Assisted Surgery
  5. 2026Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models
  6. 2026DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation
  7. 2025VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotion
  8. 2025DRAW2ACT: Turning Depth-Encoded Trajectories into Robotic Demonstration Videos