mlsanityData to 7 Oct 2026

Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation

arXiv 2201.07779 ↗First posted 19 Jan 2022cs.RO cs.CV cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

Learning to solve precision-based manipulation tasks from visual feedback using Reinforcement Learning (RL) could drastically reduce the engineering efforts required by traditional robot systems. However, performing fine-grained motor control from visual inputs alone is challenging, especially with a static third-person camera as often used in previous work. We propose a setting for robotic manipulation in which the agent receives visual feedback from both a third-person camera and an egocentric camera mounted on the robot's wrist. While the third-person camera is static, the egocentric camera enables the robot to actively control its vision to aid in precise manipulation. To fuse visual information from both cameras effectively, we additionally propose to use Transformers with a cross-view attention mechanism that models spatial attention from one view to another (and vice-versa), and use the learned features as input to an RL policy. Our method improves learning over strong single-view and multi-view baselines, and successfully transfers to a set of challenging manipulation tasks on a real robot with uncalibrated cameras, no access to state information, and a high degree of task variability. In a hammer manipulation task, our method succeeds in 75% of trials versus 38% and 13% for multi-view and single-view baselines, respectively.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 40 of its 60 references that are in the map. First posted on arXiv 19 Jan 2022.

Builds on

40 of 60 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2016Layer Normalizationcited by 5,579
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2015Continuous control with deep reinforcement learningcited by 3,279
  7. 2017Non-local Neural Networkscited by 2,290
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  10. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  11. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  12. 2018Learning Dexterous In-Hand Manipulationcited by 643
28 more
  1. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  2. 2019DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusioncited by 352
  3. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  4. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  5. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  6. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  7. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  8. 2018Residual Reinforcement Learning for Robot Controlcited by 242
  9. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  10. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
  11. 2018Learning Attentional Communication for Multi-Agent Cooperationcited by 170
  12. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 150
  13. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  14. 2019TossingBot: Learning to Throw Arbitrary Objects with Residual Physicscited by 131
  15. 2020Self-Supervised Policy Adaptation during Deploymentcited by 121
  16. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  17. 2013Deep Learning for Detecting Robotic Graspscited by 114
  18. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  19. 2020Learning Predictive Representations for Deformable Objects Using Contrastive Estimationcited by 72
  20. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  21. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 71
  22. 2018A Unified Framework for Multi-View Multi-Class Object Pose Estimationcited by 49
  23. 2021Unsupervised Learning of Visual 3D Keypoints for Controlcited by 30
  24. 2018Reinforcement Learning of Active Vision for Manipulating Objects under Occlusionscited by 25
  25. 2020Learning Precise 3D Manipulation from Multiple Uncalibrated Camerascited by 18
  26. 2020Fast Uncertainty Quantification for Deep Object Pose Estimationcited by 18
  27. 2018Learn-to-Score: Efficient 3D Scene Exploration by Predicting View Utilitycited by 15
  28. 2020Active Perception and Representation for Robotic Manipulationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 11; 2024: 12; 2025: 9; 2026: 8.

49 papers.

Sort:

By area:

  1. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  2. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  3. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priorscited by 135
  4. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 135
  5. 2023Multi-View Masked World Models for Visual Robotic Manipulationcited by 78
  6. 2023NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesiscited by 56
  7. 2022Vision-Based Manipulators Need to Also See from Their Handscited by 46
  8. 2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 45
  9. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  10. 2023Finetuning Offline World Models in the Real Worldcited by 39
  11. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 36
  12. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
37 more
  1. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  2. 2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 32
  3. 2024Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videoscited by 32
  4. 2022Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Cloudscited by 31
  5. 2022Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Spacecited by 13
  6. 2023Improving Generalization in Visual Reinforcement Learning via Conflict-aware Gradient Agreement Augmentationcited by 13
  7. 2024POV: Prompt-Oriented View-Agnostic Learning for Egocentric Hand-Object Interaction in the Multi-View Worldcited by 10
  8. 2024Multi-view Disentanglement for Reinforcement Learning with Multiple Camerascited by 8
  9. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8
  10. 2023Visual-Policy Learning through Multi-Camera View to Single-Camera View Knowledge Distillation for Robot Manipulation Taskscited by 7
  11. 2023Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrationscited by 6
  12. 2024Egocentric and Exocentric Methods: A Short Surveycited by 6
  13. 2023ImageManip: Image-based Robotic Manipulation with Affordance-guided Next View Selectioncited by 5
  14. 2024Grasp, See, and Place: Efficient Unknown Object Rearrangement with Policy Structure Priorcited by 5
  15. 2024Focus On What Matters: Separated Models For Visual-Based RL Generalizationcited by 5
  16. 2025A Practical Guide for Incorporating Symmetry in Diffusion Policycited by 4
  17. 20253D Equivariant Visuomotor Policy Learning via Spherical Projectioncited by 4
  18. 2023Dynamic Visual Prompt Tuning for Parameter Efficient Transfer Learningcited by 3
  19. 2024Articulated Object Manipulation using Online Axis Estimation with SAM2-Based Trackingcited by 3
  20. 2023Gender Bias in Transformer Models: A comprehensive surveycited by 2
  21. 2024Learning to Look: Seeking Information for Decision Making via Policy Factorizationcited by 2
  22. 2025Merging and Disentangling Views in Visual Reinforcement Learning for Robotic Manipulationcited by 2
  23. 2025New Spiking Architecture for Multi-Modal Decision-Making in Autonomous Vehiclescited by 2
  24. 2024Efficient Training of Generalizable Visuomotor Policies via Control-Aware Augmentationcited by 1
  25. 2025Learning Fused State Representations for Control from Multi-View Observationscited by 1
  26. 2025Salience-Invariant Consistent Policy Learning for Generalization in Visual Reinforcement Learningcited by 1
  27. 2026World-to-Wrist: Task-Conditioned Future Wrist Modeling for Fine-Grained Robot Manipulationcited by 1
  28. 2026AtlasVLA: Persistent World-Ego State Modeling for Vision-Language-Action Modelscited by 1
  29. 2023Knolling Bot: Teaching Robots the Human Notion of Tidinesscited by 0
  30. 2025Multimodal Fusion for Sim2real Transfer in Visual Reinforcement Learningcited by 0
  31. 2025DeGuV: Depth-Guided Visual Reinforcement Learning for Generalization and Interpretability in Manipulationcited by 0
  32. 2026VideoWeaver: Multimodal Multi-View Video-to-Video Transfer for Embodied Agentscited by 0
  33. 2026Fuzzy Encoding-Decoding to Improve Spiking Q-Learning Performance in Autonomous Drivingcited by 0
  34. 2026Integrating Causal DAGs in Deep RL: Activating Minimal Markovian States with Multi-Order Exposurecited by 0
  35. 2026RoboPIN: Grounded Embodied Reasoning via Pinned Chain-of-Thoughtcited by 0
  36. 2026Task-Relevant Representation Decoupling for Visual Reinforcement Learning Generalizationcited by 0
  37. 2026Bimanual Manipulation Within an 8 GB Budget: Zero-Copy Sensing and Quantized ACT on an Entry-Level Jetsoncited by 0