mlsanityData to 14 Oct 2026

MR-COSMO: Visual-Text Memory Recall and Direct CrOSs-MOdal Alignment Method for Query-Driven 3D Segmentation

arXiv 2506.20991 ↗First posted 26 Jun 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

The rapid advancement of vision-language models (VLMs) in 3D domains has accelerated research in text-query-guided point cloud processing, though existing methods underperform in point-level segmentation due to inadequate 3D-text alignment that limits local feature-text context linking. To address this limitation, we propose MR-COSMO, a Visual-Text Memory Recall and Direct CrOSs-MOdal Alignment Method for Query-Driven 3D Segmentation, establishing explicit alignment between 3D point clouds and text/2D image data through a dedicated direct cross-modal alignment module while implementing a visual-text memory module with specialized feature banks. This direct alignment mechanism enables precise fusion of geometric and semantic features, while the memory module employs specialized banks storing text features, visual features, and their correspondence mappings to dynamically enhance scene-specific representations via attention-based knowledge recall. Comprehensive experiments across 3D instruction, reference, and semantic segmentation benchmarks confirm state-of-the-art performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 40 references that are in the map. First posted on arXiv 26 Jun 2025.

Builds on

27 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023PartDistill: 3D Shape Part Segmentation by Vision-Language Model Distillation7 Dec
  2. 2021PointCLIP: Point Cloud Understanding by CLIP4 Dec

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  3. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  5. 2020Point Transformercited by 1,470
  6. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  7. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,205
  8. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  9. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
  10. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 748
  11. 2020PCT: Point cloud transformercited by 732
  12. 2022Point Transformer V2: Grouped Vector Attention and Partition-based Poolingcited by 388
13 more
  1. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 328
  2. 2020PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentationcited by 275
  3. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
  4. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  5. 2023Spherical Transformer for LiDAR-based 3D Recognitioncited by 125
  6. 2023Segment Any Point Cloud Sequences by Distilling Vision Foundation Modelscited by 125
  7. 20183DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentationcited by 118
  8. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 99
  9. 2023Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Trainingcited by 86
  10. 2023Efficient 3D Semantic Segmentation with Superpoint Transformercited by 57
  11. 2024OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentationcited by 44
  12. 20233D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentationcited by 36
  13. 2024RefMask3D: Language-Guided Transformer for 3D Referring Segmentationcited by 25

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025All in One: Visual-Description-Guided Unified Point Cloud Segmentation
  2. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors
  3. 2023Language-Assisted 3D Scene Understanding
  4. 2024Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understanding
  5. 20253D-PointZshotS: Geometry-Aware 3D Point Cloud Zero-Shot Semantic Segmentation Narrowing the Visual-Semantic Gap
  6. 2025PGOV3D: Open-Vocabulary 3D Semantic Segmentation with Partial-to-Global Curriculum
  7. 2023Joint Representation Learning for Text and 3D Point Cloud
  8. 2026PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Models