# 3D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Tasks

arXiv: https://arxiv.org/abs/2505.05800 · HTML: https://mlsanity.com/paper/2505.05800/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in 3D and spatial-aware VLA, one of 13 directions in Vision-language-action models.
- Builds on 55 of its 76 references that are in the map.
- First posted on arXiv 9 May 2025.

## Placed in

- Vision-language-action models / [3D and spatial-aware VLA](https://mlsanity.com/vla/3d-and-spatial-aware-vla/)

## Builds on (most cited first, 30 of 55)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model](https://mlsanity.com/paper/2501.15830/) (2025)
- [LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models](https://mlsanity.com/paper/2212.04088/) (2022)
- [CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models](https://mlsanity.com/paper/2503.22020/) (2025)
- [InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models](https://mlsanity.com/paper/2404.07191/) (2024)
- [Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2210.05714/) (2022)
- [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023)
- [An Embodied Generalist Agent in 3D World](https://mlsanity.com/paper/2311.12871/) (2023)
- [Vision-Language Foundation Models as Effective Robot Imitators](https://mlsanity.com/paper/2311.01378/) (2023)
- [Robotic Control via Embodied Chain-of-Thought Reasoning](https://mlsanity.com/paper/2407.08693/) (2024)

## Built on by (most cited first, 30 of 31)

- [Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Model](https://mlsanity.com/paper/2510.12276/) (2025)
- [GeoVLA: Empowering 3D Representations in Vision-Language-Action Models](https://mlsanity.com/paper/2508.09071/) (2025)
- [SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models](https://mlsanity.com/paper/2511.15605/) (2025)
- [SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overhead](https://mlsanity.com/paper/2512.00903/) (2025)
- [QDepth-VLA: Quantized Depth Prediction as Auxiliary Supervision for Vision-Language-Action Models](https://mlsanity.com/paper/2510.14836/) (2025)
- [Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trends](https://mlsanity.com/paper/2506.20966/) (2025)
- [Any3D-VLA: Enhancing VLA Robustness via Diverse Point Clouds](https://mlsanity.com/paper/2602.00807/) (2026)
- [VLANeXt: Recipes for Building Strong VLA Models](https://mlsanity.com/paper/2602.18532/) (2026)
- [ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Models](https://mlsanity.com/paper/2602.17951/) (2026)
- [MetricAnything: Scaling Metric Depth Pretraining with Noisy Heterogeneous Sources](https://mlsanity.com/paper/2601.22054/) (2026)
- [GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specialization](https://mlsanity.com/paper/2605.12369/) (2026)
- [OmniVLA: Physically-Grounded Multimodal VLA with Unified Multi-Sensor Perception for Robotic Manipulation](https://mlsanity.com/paper/2511.01210/) (2025)
- [Pri4R: Learning World Dynamics for Vision-Language-Action Models with Privileged 4D Representation](https://mlsanity.com/paper/2603.01549/) (2026)
- [HiVLA: A Visual-Grounded-Centric Hierarchical Embodied Manipulation System](https://mlsanity.com/paper/2604.14125/) (2026)
- [UAOR: Uncertainty-aware Observation Reinjection for Vision-Language-Action Models](https://mlsanity.com/paper/2602.18020/) (2026)
- [UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models](https://mlsanity.com/paper/2602.20231/) (2026)
- [BFA++: Hierarchical Best-Feature-Aware Token Prune for Multi-View Vision Language Action Model](https://mlsanity.com/paper/2602.20566/) (2026)
- [Learning Action Manifold with Multi-view Latent Priors for Robotic Manipulation](https://mlsanity.com/paper/2605.11832/) (2026)
- [GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation](https://mlsanity.com/paper/2605.20752/) (2026)
- [VEGA: Visual Encoder Grounding Alignment for Spatially-Aware Vision-Language-Action Models](https://mlsanity.com/paper/2605.10485/) (2026)
- [MuseVLA: An Adaptive Multimodal Sensing Vision-Language-Action Model for Robotic Manipulation](https://mlsanity.com/paper/2606.17598/) (2026)
- [DepthCache: Depth-Guided Training-Free Visual Token Merging for Vision-Language-Action Model Inference](https://mlsanity.com/paper/2603.10469/) (2026)
- [R3D: Revisiting 3D Policy Learning](https://mlsanity.com/paper/2604.15281/) (2026)
- [GeoAlign: Beyond Semantics with State-Guided Spatial Alignment in VLA Models](https://mlsanity.com/paper/2606.03240/) (2026)
- [3DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-training](https://mlsanity.com/paper/2606.04436/) (2026)
- [See like a Robot: Robot-Centric Pointmaps for VLA Models](https://mlsanity.com/paper/2607.11498/) (2026)
- [SAM3D-Guided Object-Centric Representation Alignment for Vision-Language-Action Models](https://mlsanity.com/paper/2607.25912/) (2026)
- [VistaVLA: Geometry- and Semantic-Aware 3D Gaussian-Grounded VLA for Robotic Manipulation](https://mlsanity.com/paper/2607.12356/) (2026)
- [V-Link: Recovering Lost Visual Representations in Action DiT for Vision-Language-Action Models](https://mlsanity.com/paper/2608.25308/) (2026)
- [StrataVLA: Hierarchical and Efficient 3D Geometric Grounding for Vision-Language-Action Models](https://mlsanity.com/paper/2609.26071/) (2026)

## Abstract (arXiv)

Robotic manipulation in 3D requires effective computation of N degree-of-freedom joint-space trajectories that enable precise and robust control. To achieve this, robots must integrate semantic understanding with visual perception to transform real-world observations into low-level control for object interaction. Recent advances in Vision-Language-Action (VLA) models have shown promise by mapping RGB images and language instructions to task space velocities, typically trained on large datasets of teleoperated demonstrations. However, these models often struggle with generalization beyond their training distributions. In this work, we introduce 3D-CAVLA, a novel finetuning framework that enhances task generalization of VLA policies by incorporating three key components: (i) chain-of-thought reasoning for structured decision-making, (ii) depth-aware perception for 3D spatial understanding, and (iii) task-oriented region-of-interest detection for focused manipulation. Extensive experiments in the LIBERO simulation environment demonstrate that 3D-CAVLA achieves an average success rate of 98.1% across diverse in-domain task suites. On unseen tasks, 3D-CAVLA delivers an absolute improvement of 8.8% in success rate, underscoring the benefits of 3D scene awareness for robust generalization. We validate our approach on real-world tabletop experiments demonstrating that the proposed model translates effectively from simulation to physical robots. 3D-CAVLA achieves over a 3X faster training convergence and delivers a 25% gain in success rate on unseen real world tasks. We will open-source our code and the unseen tasks dataset to promote community-driven research here: https://3d-cavla.github.io