mlsanityData to 7 Oct 2026

SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representations

arXiv 2112.04680 ↗First posted 9 Dec 2021cs.CVcited by 55

Not on the map. Why?

Abstract (arXiv)

Pre-training has become a standard paradigm in many computer vision tasks. However, most of the methods are generally designed on the RGB image domain. Due to the discrepancy between the two-dimensional image plane and the three-dimensional space, such pre-trained models fail to perceive spatial information and serve as sub-optimal solutions for 3D-related tasks. To bridge this gap, we aim to learn a spatial-aware visual representation that can describe the three-dimensional space and is more suitable and effective for these tasks. To leverage point clouds, which are much more superior in providing spatial information compared to images, we propose a simple yet effective 2D Image and 3D Point cloud Unsupervised pre-training strategy, called SimIPU. Specifically, we develop a multi-modal contrastive learning framework that consists of an intra-modal spatial perception module to learn a spatial-aware representation from point clouds and an inter-modal feature interaction module to transfer the capability of perceiving spatial information from the point cloud encoder to the image encoder, respectively. Positive pairs for contrastive losses are established by the matching algorithm and the projection matrix. The whole framework is trained in an unsupervised end-to-end fashion. To the best of our knowledge, this is the first study to explore contrastive learning pre-training strategies for outdoor multi-modal datasets, containing paired camera images and LIDAR point clouds. Codes and models are available at https://github.com/zhyever/SimIPU.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 34 of its 41 references that are in the map. First posted on arXiv 9 Dec 2021.

Builds on

34 of 41 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  9. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  10. 2018Deep contextualized word representationscited by 2,285
  11. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  12. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
22 more
  1. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  2. 2016Colorful Image Colorizationcited by 1,113
  3. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 721
  4. 2020AdaBins: Depth Estimation using Adaptive Binscited by 596
  5. 2020A Survey on Contrastive Self-supervised Learningcited by 553
  6. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  7. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  8. 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 444
  9. 20203DSSD: Point-based 3D Single Stage Object Detectorcited by 440
  10. 2020Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contextscited by 210
  11. 2020Multi-Task Multi-Sensor Fusion for 3D Object Detectioncited by 203
  12. 2019MVX-Net: Multimodal VoxelNet for 3D Object Detectioncited by 162
  13. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 122
  14. 2020HR-Depth: High Resolution Self-Supervised Monocular Depth Estimationcited by 115
  15. 2015Learning to See by Movingcited by 101
  16. 2021Pri3D: Can 3D Priors Help 2D Representation Learning?cited by 61
  17. 2021Self-Supervised Pillar Motion Learning for Autonomous Drivingcited by 44
  18. 2015Learning image representations tied to ego-motioncited by 41
  19. 2021Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretrainingcited by 32
  20. 2021CoCoNets: Continuous Contrastive 3D Scene Representationscited by 19
  21. 2020Multi-View Adaptive Fusion Network for 3D Object Detectioncited by 6
  22. 2019Visuomotor Understanding for Representation Learning of Driving Scenescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 18; 2024: 14; 2025: 6; 2026: 2.

55 papers.

Sort:

By area:

  1. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 143
  2. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  3. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 121
  4. 2022DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimationcited by 116
  5. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  6. 2022ALSO: Automotive Lidar Self-supervision by Occupancy estimationcited by 68
  7. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  8. 2023PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detectioncited by 53
  9. 2023RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptionscited by 47
  10. 2022AutoAlignV2: Deformable Feature Aggregation for Dynamic Multi-Modal 3D Object Detectioncited by 46
  11. 2022Graph-DETR3D: Rethinking Overlapping Regions for Multi-View 3D Object Detectioncited by 35
  12. 2022LidarCLIP or: How I Learned to Talk to Point Cloudscited by 33
38 more
  1. 2024Towards High-resolution 3D Anomaly Detection via Group-Level Feature Contrastive Learningcited by 25
  2. 2022Let Images Give You More:Point Cloud Cross-Modal Training for Shape Analysiscited by 24
  3. 2023BEVContrast: Self-Supervision in BEV Space for Automotive Lidar Point Cloudscited by 24
  4. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  5. 2022Unsupervised Domain Adaptation for Monocular 3D Object Detection via Self-Trainingcited by 22
  6. 2023Self-supervised Learning for Pre-Training 3D Point Clouds: A Surveycited by 21
  7. 2023EVEN: An Event-Based Framework for Monocular Depth Estimation at Adverse Night Conditionscited by 19
  8. 2023Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-trainingcited by 17
  9. 2024VisionPAD: A Vision-Centric Pre-training Paradigm for Autonomous Drivingcited by 15
  10. 2023CALICO: Self-Supervised Camera-LiDAR Contrastive Pre-training for BEV Perceptioncited by 14
  11. 2023PointMBF: A Multi-scale Bidirectional Fusion Network for Unsupervised RGB-D Point Cloud Registrationcited by 14
  12. 2023The RoboDepth Challenge: Methods and Advancements Towards Robust Depth Estimationcited by 13
  13. 2023Syn-Mediverse: A Multimodal Synthetic Dataset for Intelligent Scene Understanding of Healthcare Facilitiescited by 10
  14. 2023DDOS: The Drone Depth and Obstacle Segmentation Datasetcited by 10
  15. 2024Shelf-Supervised Cross-Modal Pre-Training for 3D Object Detectioncited by 9
  16. 2024Advancing 3D Point Cloud Understanding through Deep Transfer Learning: A Comprehensive Surveycited by 9
  17. 2022Self-Supervised Learning with Multi-View Rendering for 3D Point Cloud Analysiscited by 8
  18. 2024Cross-spectral Gated-RGB Stereo Depth Estimationcited by 8
  19. 2023Gated Stereo: Joint Depth Estimation from Gated and Wide-Baseline Active Stereo Cuescited by 7
  20. 2024Self-supervised Learning of LiDAR 3D Point Clouds via 2D-3D Neural Calibrationcited by 7
  21. 2022Language-Assisted 3D Feature Learning for Semantic Scene Understandingcited by 6
  22. 2022Self-Supervised Pre-training of 3D Point Cloud Networks with Image Datacited by 5
  23. 20243D Annotation-Free Learning by Distilling 2D Open-Vocabulary Segmentation Models for Autonomous Drivingcited by 5
  24. 2024S3PT: Scene Semantics and Structure Guided Clustering to Boost Self-Supervised Pre-Training for Autonomous Drivingcited by 4
  25. 2025SQS: Enhancing Sparse Perception Models via Query-based Splatting in Autonomous Drivingcited by 4
  26. 2025Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systemscited by 4
  27. 2023Point Clouds Are Specialized Images: A Knowledge Transfer Approach for 3D Understandingcited by 3
  28. 2024Gated Fields: Learning Scene Reconstruction from Gated Videoscited by 3
  29. 2023Dyna-DepthFormer: Multi-frame Transformer for Self-Supervised Depth Estimation in Dynamic Scenescited by 2
  30. 2023Contrastive Learning for Self-Supervised Pre-Training of Point Cloud Segmentation Networks With Image Datacited by 2
  31. 2024FedRSU: Federated Learning for Scene Flow Estimation on Roadside Unitscited by 2
  32. 2022LiteDepth: Digging into Fast and Accurate Depth Estimation on Mobile Devicescited by 1
  33. 2024Learning Shared RGB-D Fields: Unified Self-supervised Pre-training for Label-efficient LiDAR-Camera 3D Perceptioncited by 1
  34. 2024CLAP: Unsupervised 3D Representation Learning for Fusion 3D Perception via Curvature Sampling and Prototype Learningcited by 1
  35. 20252D-3D Attention and Entropy for Pose Robust 2D Facial Recognitioncited by 1
  36. 2022How do Cross-View and Cross-Modal Alignment Affect Representations in Contrastive Learning?cited by 0
  37. 2023Towards Transferable Multi-modal Perception Representation Learning for Autonomy: NeRF-Supervised Masked AutoEncodercited by 0
  38. 2024P3P: Pseudo-3D Pre-training for Scaling 3D Voxel-based Masked Autoencoderscited by 0

50 most cited of 55 All 55 (JSON)