Spatially Visual Perception for End-to-End Robotic Learning
On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning
Abstract (arXiv)
Recent advances in imitation learning have shown significant promise for robotic control and embodied intelligence. However, achieving robust generalization across diverse mounted camera observations remains a critical challenge. In this paper, we introduce a video-based spatial perception framework that leverages 3D spatial representations to address environmental variability, with a focus on handling lighting changes. Our approach integrates a novel image augmentation technique, AugBlender, with a state-of-the-art monocular depth estimation model trained on internet-scale data. Together, these components form a cohesive system designed to enhance robustness and adaptability in dynamic scenarios. Our results demonstrate that our approach significantly boosts the success rate across diverse camera exposures, where previous models experience performance collapse. Our findings highlight the potential of video-based spatial perception models in advancing robustness for end-to-end robotic learning, paving the way for scalable, low-cost solutions in embodied intelligence.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 48 references that are in the map. First posted on arXiv 26 Nov 2024.
Builds on
37 of 48 references.
Closest ancestors · 2
Same direction, newest first
- 2024Data Scaling Laws in Imitation Learning for Robotic Manipulation24 Oct
- 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation4 Jan
Other references · 35
- 2017Attention Is All You Needcited by 59,932
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
- 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
- 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
- 2016End to End Learning for Self-Driving Carscited by 1,000
- 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
- 2019Point-Voxel CNN for Efficient 3D Deep Learningcited by 343
23 more
- 2019Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Comingcited by 332
- 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 306
- 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
- 2023MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimationcited by 173
- 2022Understanding The Robustness in Vision Transformerscited by 144
- 2023Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Drivingcited by 112
- 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 106
- 2024ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigationcited by 67
- 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 62
- 20204Seasons: A Cross-Season Dataset for Multi-Weather SLAM in Autonomous Drivingcited by 51
- 2021Cross-domain Imitation from Observationscited by 42
- 2023Enhancing Visibility in Nighttime Haze Images Using Guided APSF and Gradient Adaptive Convolutioncited by 39
- 2023AutoFed: Heterogeneity-Aware Federated Multimodal Learning for Robust Autonomous Drivingcited by 32
- 20232PCNet: Two-Phase Consistency Training for Day-to-Night Unsupervised Domain Adaptive Object Detectioncited by 30
- 2020Robust Imitation Learning from Noisy Demonstrationscited by 26
- 2021Safe Occlusion-aware Autonomous Driving via Game-Theoretic Active Perceptioncited by 22
- 2023Imitation with Spatial-Temporal Heatmap: 2nd Place Solution for NuPlan Challengecited by 19
- 2023End-To-End Planning of Autonomous Driving in Industry and Academia: 2022-2023cited by 11
- 2024PrimeDepth: Efficient Monocular Depth Estimation with a Stable Diffusion Preimagecited by 11
- 2022Robust Imitation Learning from Corrupted Demonstrationscited by 10
- 2024Visually Robust Adversarial Imitation Learning from Videos with Contrastive Learningcited by 9
- 2023Occupancy Prediction-Guided Neural Planner for Autonomous Drivingcited by 8
- 2023Robust Visual Imitation Learning with Inverse Dynamics Representationscited by 8
Built on by
Citing papers per year: 2025: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Spatial-Temporal Aware Visuomotor Diffusion Policy Learning
- 2026SpatialVAM:Spatial-Aware Multi-View Video Diffusion as a Data-Efficient Robot Policy
- 2025Improving Robotic Manipulation with Efficient Geometry-Aware Vision Encoder
- 2026WSA1: a 3D-Centric World-Spatial-Action Model for Generalizable Robot Control
- 2026StereoPatch: Patch-Aligned RGB-Depth Fusion for Spatial Perception in Robot Manipulation
- 2025VGGT-DP: Generalizable Robot Control via Vision Foundation Models
- 2025Spatial Policy: Guiding Visuomotor Robotic Manipulation with Spatial-Aware Modeling and Reasoning
- 2025TesserAct: Learning 4D Embodied World Models