mlsanityData to 14 Oct 2026

PARIS3D: Reasoning-based 3D Part Segmentation Using Large Multimodal Model

arXiv 2404.03836 ↗First posted 4 Apr 2024cs.CV cs.AIcited by 9

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Recent advancements in 3D perception systems have significantly improved their ability to perform visual recognition tasks such as segmentation. However, these systems still heavily rely on explicit human instruction to identify target objects or categories, lacking the capability to actively reason and comprehend implicit user intentions. We introduce a novel segmentation task known as reasoning part segmentation for 3D objects, aiming to output a segmentation mask based on complex and implicit textual queries about specific parts of a 3D object. To facilitate evaluation and benchmarking, we present a large 3D dataset comprising over 60k instructions paired with corresponding ground-truth part segmentation annotations specifically curated for reasoning-based 3D part segmentation. We propose a model that is capable of segmenting parts of 3D objects based on implicit textual queries and generating natural language explanations corresponding to 3D object segmentation requests. Experiments show that our method achieves competitive performance to models that use explicit queries, with the additional abilities to identify part concepts, reason about them, and complement them with world knowledge. Our source code, dataset, and trained models are available at https://github.com/AmrinKareem/PARIS3D.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 52 references that are in the map. First posted on arXiv 4 Apr 2024.

Builds on

47 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov

Other references · 46

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  9. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  10. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,205
  11. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  12. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
34 more
  1. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  2. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 784
  3. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  4. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 748
  5. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  6. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  7. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 436
  8. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  9. 2017Large-scale Point Cloud Semantic Segmentation with Superpoint Graphscited by 385
  10. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  11. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  12. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  13. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  14. 2017Submanifold Sparse Convolutional Networkscited by 223
  15. 2021PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Cloudscited by 211
  16. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  17. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 200
  18. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  19. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  20. 2022Stratified Transformer for 3D Point Cloud Segmentationcited by 167
  21. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 135
  22. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  23. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 129
  24. 2022PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Modelscited by 118
  25. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  26. 2020Virtual Multi-view Fusion for 3D Semantic Segmentationcited by 99
  27. 2018Recurrent Slice Networks for 3D Segmentation of Point Cloudscited by 82
  28. 2020Few-shot 3D Point Cloud Semantic Segmentationcited by 77
  29. 2023ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidancecited by 70
  30. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60
  31. 2022Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentationcited by 47
  32. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Promptscited by 32
  33. 2020Label-Efficient Learning on Point Clouds using Approximate Convex Decompositionscited by 27
  34. 20223D Concept Grounding on Neural Fieldscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 4.

9 papers.

  1. 2025MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentationcited by 16
  2. 2024Kestrel: 3D Multimodal LLM for Part-Aware Grounded Descriptioncited by 9
  3. 2024Multimodal 3D Reasoning Segmentation with Complex Scenescited by 8
  4. 2025AdaToken-3D: Dynamic Spatial Gating for Efficient 3D Large Multimodal-Models Reasoningcited by 5
  5. 2026DreamPartGen: Semantically Grounded Part-Level 3D Generation via Collaborative Latent Denoisingcited by 2
  6. 2026GeoHeight-Bench: Towards Height-Aware Multimodal Reasoning in Remote Sensingcited by 1
  7. 2025PointCubeNet: 3D Part-level Reasoning with 3x3x3 Point Cloud Blockscited by 0
  8. 20263D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Modelscited by 0
  9. 2026MV-GEL: Language-Driven Multi-View Geometric Entity Localization on Meshescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Reasoning3D -- Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-Language Models
  2. 2024Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model
  3. 2025OpenMaskDINO3D : Reasoning 3D Segmentation via Large Language Model
  4. 2026PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understanding
  5. 20243D Part Segmentation via Geometric Aggregation of 2D Visual Features
  6. 2025Enhancing Spatial Reasoning in Multimodal Large Language Models through Reasoning-based Segmentation
  7. 2024SAMPart3D: Segment Any Part in 3D Objects
  8. 2026SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes