mlsanityData to 14 Oct 2026

Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learning

arXiv 2407.15815 ↗First posted 22 Jul 2024cs.RO cs.AI cs.CVcited by 41

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

Can we endow visuomotor robots with generalization capabilities to operate in diverse open-world scenarios? In this paper, we propose Maniwhere, a generalizable framework tailored for visual reinforcement learning, enabling the trained robot policies to generalize across a combination of multiple visual disturbance types. Specifically, we introduce a multi-view representation learning approach fused with Spatial Transformer Network (STN) module to capture shared semantic information and correspondences among different viewpoints. In addition, we employ a curriculum-based randomization and augmentation approach to stabilize the RL training process and strengthen the visual generalization ability. To exhibit the effectiveness of Maniwhere, we meticulously design 8 tasks encompassing articulate objects, bi-manual, and dexterous hand manipulation tasks, demonstrating Maniwhere's strong visual generalization and sim2real transfer abilities across 3 hardware platforms. Our experiments show that Maniwhere significantly outperforms existing state-of-the-art methods. Videos are provided at https://gemcollector.github.io/maniwhere/.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 56 of its 60 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

56 of 60 references.

References · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  8. 2015Spatial Transformer Networkscited by 1,429
  9. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  10. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  11. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  12. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 301
38 more
  1. 2022Masked Visual Pre-training for Motor Controlcited by 248
  2. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 225
  3. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
  4. 2023Language-Driven Representation Learning for Roboticscited by 165
  5. 2019Temporal Cycle-Consistency Learningcited by 154
  6. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 149
  7. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 137
  8. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 133
  9. 2020Self-Supervised Policy Adaptation during Deploymentcited by 123
  10. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 107
  11. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 105
  12. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  13. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 82
  14. 2023Multi-View Masked World Models for Visual Robotic Manipulationcited by 80
  15. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 79
  16. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 71
  17. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 60
  18. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 59
  19. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 57
  20. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 56
  21. 2023Robot Learning with Sensorimotor Pre-trainingcited by 51
  22. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 50
  23. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  24. 2023Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulationcited by 49
  25. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  26. 2024Learning Manipulation by Predicting Interactioncited by 37
  27. 2023Dynamic Handover: Throw and Catch with Bimanual Handscited by 35
  28. 2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 34
  29. 2024SPIN: Simultaneous Perception, Interaction and Navigationcited by 29
  30. 2023RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalizationcited by 28
  31. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 28
  32. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 25
  33. 2023For Pre-Trained Vision Models in Motor Control, Not All Policy Learning Methods are Created Equalcited by 22
  34. 2022Don't Touch What Matters: Task-Aware Lipschitz Data Augmentation for Visual Reinforcement Learningcited by 21
  35. 2024Natural Language Can Help Bridge the Sim2Real Gapcited by 21
  36. 2023MoVie: Visual Model-Based Policy Adaptation for View Generalizationcited by 16
  37. 2023Normalization Enhances Generalization in Visual Reinforcement Learningcited by 13
  38. 2023Generalizable Visual Reinforcement Learning with Segment Anything Modelcited by 12

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 20; 2026: 13.

41 papers.

Sort:

By area:

  1. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 151
  2. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 89
  3. 2024SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splattingcited by 69
  4. 2024Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoningcited by 57
  5. 2024Generalizable Humanoid Manipulation with 3D Diffusion Policiescited by 52
  6. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  7. 2025A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Modelscited by 38
  8. 2024Run-time Observation Interventions Make Vision-Language-Action Models More Visually Robustcited by 33
  9. 2025RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Modelscited by 28
  10. 2025H3DP: Triply-Hierarchical Diffusion Policy for Visuomotor Learningcited by 16
  11. 2025Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfercited by 15
  12. 2024Effective Tuning Strategies for Generalist Robot Manipulation Policiescited by 10
29 more
  1. 2025Generalizable Domain Adaptation for Sim-and-Real Policy Co-Trainingcited by 10
  2. 2025Tool-as-Interface: Learning Robot Policies from Observing Human Tool Usecited by 8
  3. 2025Robot Trains Robot: Automatic Real-World Policy Adaptation and Learning for Humanoidscited by 7
  4. 2025AnyBody: A Benchmark Suite for Cross-Embodiment Manipulationcited by 6
  5. 2025ManiVID-3D: Generalizable View-Invariant Reinforcement Learning for Robotic Manipulation via Disentangled 3D Representationscited by 5
  6. 2026ViTaS: Visual Tactile Soft Fusion Contrastive Learning for Visuomotor Learningcited by 5
  7. 2024ManiBox: Enhancing Embodied Spatial Generalization via Scalable Simulation Data Generationscited by 4
  8. 2024Stem-OB: Generalizable Visual Imitation Learning with Stem-Like Convergent Observation through Diffusion Inversioncited by 4
  9. 2025Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agentcited by 4
  10. 2025Bridging the Sim2Real Gap: Vision Encoder Pre-Training for Visuomotor Policy Transfercited by 3
  11. 2025Wheeled Lab: Modern Sim2Real for Low-cost, Open-source Wheeled Roboticscited by 3
  12. 2025Merging and Disentangling Views in Visual Reinforcement Learning for Robotic Manipulationcited by 2
  13. 2026Squint: Fast Visual Reinforcement Learning for Sim-to-Real Roboticscited by 2
  14. 2026DreamSAC: Learning Hamiltonian World Models via Symmetry Explorationcited by 1
  15. 20263D Generation for Embodied AI and Robotic Simulation: A Surveycited by 1
  16. 2026HALO-WA: Hybrid-Attention Latent-Guided Online Reinforcement Learning for World-Action Modelscited by 1
  17. 2024FunGrasp: Functional Grasping for Diverse Dexterous Handscited by 0
  18. 2025A Visual Reinforcement Learning-Based Separate Primitive Policy for Peg-in-Hole Taskscited by 0
  19. 2025RoboPearls: Editable Video Simulation for Robot Manipulationcited by 0
  20. 2025Multimodal Fusion for Sim2real Transfer in Visual Reinforcement Learningcited by 0
  21. 2025DeGuV: Depth-Guided Visual Reinforcement Learning for Generalization and Interpretability in Manipulationcited by 0
  22. 2026Efficient Unsupervised Environment Design through Hierarchical Policy Representation Learningcited by 0
  23. 2026TransDex: Pre-training Visuo-Tactile Policy with Point Cloud Reconstruction for Dexterous Manipulation of Transparent Objectscited by 0
  24. 2026HeteroGenManip: Generalizable Manipulation For Heterogeneous Object Interactionscited by 0
  25. 2026Scalable Multi-Task Data Generation via Reinforcement Learning for Language-Conditioned Bimanual Dexterous Manipulationcited by 0
  26. 2026AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learningcited by 0
  27. 2026UniPart: Towards Zero-shot Language-Grounded 3D Part Segmentation for Embodied Interactioncited by 0
  28. 2026An Empirical Study on What Matters for Viewpoint-Generalizable Policies in Visual Imitation Learningcited by 0
  29. 2026VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Zero-Shot Visual Generalization in Robot Manipulation
  2. 2026Inductive Generalization for Robotic Manipulation
  3. 2025ObjectVLA: End-to-End Open-World Object Manipulation Without Demonstration
  4. 2025VGGT-DP: Generalizable Robot Control via Vision Foundation Models
  5. 2023Visual-Policy Learning through Multi-Camera View to Single-Camera View Knowledge Distillation for Robot Manipulation Tasks
  6. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generation
  7. 2025ManipLVM-R1: Reinforcement Learning for Reasoning in Embodied Manipulation with Large Vision-Language Models
  8. 2025VideoVLA: Video Generators Can Be Generalizable Robot Manipulators