mlsanityData to 7 Oct 2026

RayViT: Ray-Conditioned Visual Representations for Viewpoint-Robust Imitation Learning

arXiv 2607.29622 ↗First posted 31 Jul 2026cs.RO cs.CVcited by 0

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Visual imitation learning enables robots to acquire visuomotor skills directly from images, yet RGB observations lack explicit geometric cues, making learned policies brittle to camera perturbations. To address this, we propose Ray-conditioned Vision Transformer Encoder (RayViT), a lightweight architecture that injects camera geometry into pretrained ViT backbones. RayViT represents camera geometry as a Pl"ucker ray map, patchifies it into ray features, and uses gated cross-attention to produce a ray-conditioned class token. These ray features are added as dense positional embeddings, while the ray class token replaces the original ViT class token to provide a geometry-aware summary representation. We combine this approach with an auxiliary cosine similarity loss to consistently improve the performance and robustness for geometry-aware tokens. Experiments on sim- and real-robot tasks demonstrate that RayViT improves robustness by approximately 13 percentage points under camera perturbations in multi-task RoboCasa benchmark and by 1.78 average completed stages in real-world multi-task success rate compared to baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 42 references that are in the map. First posted on arXiv 31 Jul 2026.

Builds on

37 of 42 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Fourier Features Let Agents Learn High Precision Policies with Imitation Learning10 Jun
  2. 2025PointMapPolicy: Structured Point Cloud Processing for Multi-Modal Imitation Learning23 Oct
  3. 2025Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioning2 Oct
  4. 2025Improving Robotic Manipulation with Efficient Geometry-Aware Vision Encoder19 Sep
  5. 2025Canonical Policy: Learning Canonical 3D Representation for SE(3)-Equivariant Policy24 May
  6. 2025Adapt3R: Adaptive 3D Scene Representation for Domain Transfer in Imitation Learning6 Mar
  7. 2023Learning Generalizable Manipulation Policies with Object-Centric 3D Representations22 Oct

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Segment Anythingcited by 8,158
  3. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  5. 2025DINOv3cited by 1,297
  6. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  7. 2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 626
  8. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  9. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  10. 2020Reinforcement Learning with Augmented Datacited by 452
  11. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  12. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 330
18 more
  1. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  2. 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Modelscited by 278
  3. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  4. 2022Masked Visual Pre-training for Motor Controlcited by 246
  5. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  6. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  7. 2023Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulationcited by 135
  8. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 135
  9. 2025Aether: Geometric-Aware Unified World Modelingcited by 94
  10. 2022SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fieldscited by 51
  11. 2024RoVi-Aug: Robot and Viewpoint Augmentation for Cross-Embodiment Robot Learningcited by 46
  12. 2024View-Invariant Policy Learning via Zero-Shot Novel View Synthesiscited by 35
  13. 2025GeoAware-VLA: Implicit Geometry Aware Vision-Language-Action Modelcited by 24
  14. 2025CLASS: Contrastive Learning via Action Sequence Supervision for Robot Manipulationcited by 10
  15. 2025EquAct: An SE(3)-Equivariant Multi-Task Transformer for Open-Loop Robotic Manipulationcited by 8
  16. 2026Efficient Universal Perception Encodercited by 6
  17. 2026LagerNVS: Latent Geometry for Fully Neural Real-time Novel View Synthesiscited by 4
  18. 2026RayMap3R: Inference-Time RayMap for Dynamic 3D Reconstructioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025VGGT-DP: Generalizable Robot Control via Vision Foundation Models
  2. 2020A Geometric Perspective on Visual Imitation Learning
  3. 2021The Surprising Effectiveness of Representation Learning for Visual Imitation
  4. 2024Spatially Visual Perception for End-to-End Robotic Learning
  5. 2026More Structure, Not More Capacity: Object-Centric Representations for Visuomotor Imitation Learning
  6. 2026Multi-Camera View Scaling for Data-Efficient Robot Imitation Learning
  7. 2026An Empirical Study on What Matters for Viewpoint-Generalizable Policies in Visual Imitation Learning
  8. 2021Towards More Generalizable One-shot Visual Imitation Learning