mlsanityData to 14 Oct 2026

CrossOver: 3D Scene Cross-Modal Alignment

arXiv 2502.15011 ↗First posted 20 Feb 2025cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Multi-modal 3D object understanding has gained significant attention, yet current approaches often assume complete data availability and rigid alignment across all modalities. We present CrossOver, a novel framework for cross-modal 3D scene understanding via flexible, scene-level modality alignment. Unlike traditional methods that require aligned modality data for every object instance, CrossOver learns a unified, modality-agnostic embedding space for scenes by aligning modalities -- RGB images, point clouds, CAD models, floorplans, and text descriptions -- with relaxed constraints and without explicit object semantics. Leveraging dimensionality-specific encoders, a multi-stage training pipeline, and emergent cross-modal behaviors, CrossOver supports robust scene retrieval and object localization, even with missing modalities. Evaluations on ScanNet and 3RScan datasets show its superior performance across diverse metrics, highlighting the adaptability for real-world applications in 3D scene understanding.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 37 of its 44 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

37 of 44 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  5. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  6. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,443
  7. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  8. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  9. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  10. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  11. 2015NetVLAD: CNN architecture for weakly supervised place recognitioncited by 941
  12. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
25 more
  1. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  2. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  3. 2021Vector Neurons: A General Framework for SO(3)-Equivariant Networkscited by 305
  4. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 281
  5. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 280
  6. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 208
  7. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  8. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 202
  9. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  10. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 188
  11. 2018Learning Factorized Multimodal Representationscited by 171
  12. 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 144
  13. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 144
  14. 2018Scan2CAD: Learning CAD Model Alignment in RGB-D Scanscited by 142
  15. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  16. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 122
  17. 20193-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agentscited by 74
  18. 2022Text2Pos: Text-to-Point-Cloud Cross-Modal Localizationcited by 40
  19. 2022LidarCLIP or: How I Learned to Talk to Point Cloudscited by 33
  20. 2023SGAligner : 3D Scene Alignment with Scene Graphscited by 30
  21. 2024SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphscited by 29
  22. 2023LIP-Loc: LiDAR Image Pretraining for Cross-Modal Localizationcited by 26
  23. 2021Mending Neural Implicit Modeling for 3D Vehicle Reconstruction in the Wildcited by 19
  24. 2023Living Scenes: Multi-object Relocalization and Reconstruction in Changing 3D Environmentscited by 15
  25. 2024SG-PGM: Partial Graph Matching Network with Semantic Geometric Fusion for 3D Scene Graph Alignment and Its Downstream Taskscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 8.

12 papers.

  1. 2025SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splattingcited by 15
  2. 20253DAlign-DAER: Dynamic Attention Policy and Efficient Retrieval Strategy for Fine-grained 3D-Text Alignment at Scalecited by 12
  3. 2025SGAligner++: Cross-Modal Language-Aided 3D Scene Graph Alignmentcited by 2
  4. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  5. 2026Uncertainty-guided Compositional Alignment with Part-to-Whole Semantic Representativeness in Hyperbolic Vision-Language Modelscited by 2
  6. 2025Object-X: Learning to Reconstruct Multi-Modal 3D Object Representationscited by 1
  7. 2026PartRAG: Retrieval-Augmented Part-Level 3D Generation and Editingcited by 1
  8. 2026PlanaReLoc: Camera Relocalization in 3D Planar Primitives via Region-Based Structure Matchingcited by 0
  9. 2026RGB-Pointmap Pretraining for Unified 3D Scene Understandingcited by 0
  10. 2026HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understandingcited by 0
  11. 2026Multimodal Floorplan Encoding: Learning Dense Modality-Invariant Representationscited by 0
  12. 2026From Alignment to Fusion in 3D Vision-Languagecited by 0