mlsanityData to 14 Oct 2026

Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulation

arXiv 2411.18623 ↗First posted 27 Nov 2024cs.CVcited by 46

Not on the map. Why?

Cited by 22 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

3D geometric information is essential for manipulation tasks, as robots need to perceive the 3D environment, reason about spatial relationships, and interact with intricate spatial configurations. Recent research has increasingly focused on the explicit extraction of 3D features, while still facing challenges such as the lack of large-scale robotic 3D data and the potential loss of spatial geometry. To address these limitations, we propose the Lift3D framework, which progressively enhances 2D foundation models with implicit and explicit 3D robotic representations to construct a robust 3D manipulation policy. Specifically, we first design a task-aware masked autoencoder that masks task-relevant affordance patches and reconstructs depth information, enhancing the 2D foundation model's implicit 3D robotic representation. After self-supervised fine-tuning, we introduce a 2D model-lifting strategy that establishes a positional mapping between the input 3D points and the positional embeddings of the 2D model. Based on the mapping, Lift3D utilizes the 2D foundation model to directly encode point cloud data, leveraging large-scale pretrained knowledge to construct explicit 3D robotic representations while minimizing spatial information loss. In experiments, Lift3D consistently outperforms previous state-of-the-art methods across several simulation benchmarks and real-world scenarios.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 83 of its 95 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

83 of 95 references.

References · 83

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  9. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  12. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
38 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  6. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  9. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  10. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  11. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  12. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  13. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  14. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  15. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  16. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  17. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  18. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  19. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  20. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 436
  21. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  22. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 394
  23. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  24. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 349
  25. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
  26. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 282
  27. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 225
  28. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  29. 2022AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domainscited by 208
  30. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  31. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  32. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 182
  33. 2022Masked World Models for Visual Controlcited by 170
  34. 2023Language-Driven Representation Learning for Roboticscited by 165
  35. 2023ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulationcited by 162
  36. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 144
  37. 2023Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulationcited by 138
  38. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 137

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 31; 2026: 13.

46 papers.

Sort:

By area:

  1. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 181
  2. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 125
  3. 2025PointVLA: Injecting the 3D World into Vision-Language-Action Modelscited by 105
  4. 2025BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Modelscited by 80
  5. 2025MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulationcited by 70
  6. 2025GeoVLA: Empowering 3D Representations in Vision-Language-Action Modelscited by 70
  7. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 68
  8. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 63
  9. 2025Evo-0: Vision-Language-Action Model with Implicit Spatial Understandingcited by 58
  10. 2025ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledgecited by 49
  11. 2025DepthVLA: Enhancing Vision-Language-Action Models with Depth-Aware Spatial Reasoningcited by 46
  12. 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 40
34 more
  1. 2025DeepVerse: 4D Autoregressive Video Generation as a World Modelcited by 40
  2. 2024AffordDP: Generalizable Diffusion Policy with Transferable Affordancecited by 37
  3. 2025Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoningcited by 31
  4. 2025MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoderscited by 26
  5. 2025SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overheadcited by 26
  6. 2025AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulationcited by 23
  7. 2025SciVerse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMMs on Multi-modal Scientific Problemscited by 22
  8. 2025ActiveUMI: Robotic Manipulation with Active Perception from Robot-Free Human Demonstrationscited by 19
  9. 2025CordViP: Correspondence-based Visuomotor Policy for Dexterous Manipulation in Real-Worldcited by 18
  10. 2025AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletonscited by 18
  11. 2025StereoVLA: Enhancing Vision-Language-Action Models with Stereo Visioncited by 18
  12. 2026ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulationcited by 18
  13. 2025GP3: A 3D Geometry-Aware Policy with Multi-View Images for Robotic Manipulationcited by 12
  14. 2025Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Compositioncited by 9
  15. 2025CrayonRobo: Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulationcited by 6
  16. 2026ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulationcited by 6
  17. 2025SpatialActor: Exploring Disentangled Spatial Representations for Robust Robotic Manipulationcited by 5
  18. 2025ADPro: a Test-time Adaptive Diffusion Policy via Manifold-constrained Denoising and Task-aware Initialization for Robotic Manipulationcited by 4
  19. 2026Understanding the Impact of Geometric Foundation Models on Vision-Language-Action Modelscited by 4
  20. 2026DeVA: Decoupled Video-Action Model with physical guidance for robot policy learningcited by 4
  21. 2025Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challengescited by 3
  22. 2026OASIS: Observation-Action Space Alignment via SE(3) Trajectory Prediction for Robotic Manipulationcited by 3
  23. 2026From Fixed to Free Cameras: Calibration-Free View-Robust Vision-Language-Action Modelcited by 3
  24. 2025Avi: Action from Volumetric Inferencecited by 2
  25. 2026From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulationcited by 2
  26. 2025SR3D: Unleashing Single-view 3D Reconstruction for Transparent and Specular Object Graspingcited by 1
  27. 2026R3D: Revisiting 3D Policy Learningcited by 1
  28. 2026Learning Structural Latent Points for Efficient Visual Representations in Robotic Manipulationcited by 1
  29. 20263DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-trainingcited by 1
  30. 2026ExStereo: Lifting 2D Vision-Language-Action Models to 3D with Explicit Stereo Representationscited by 1
  31. 20253DWG: 3D Weakly Supervised Visual Grounding via Category and Instance-Level Alignmentcited by 0
  32. 2025No Need for Real 3D: Fusing 2D Vision with Pseudo 3D Representations for Robotic Manipulation Learningcited by 0
  33. 2026Attention-DP3: Spatially Object-aware 3D Diffusion Policy via Geometry-aligned Attentional Conditioningcited by 0
  34. 2026GeomVLA: Unifying Scene, Motion, and Action in 3Dcited by 0