Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Model
Not on the map. Why?
Abstract (arXiv)
3D object segmentation with Large Language Models (LLMs) has become a prevailing paradigm due to its broad semantics, task flexibility, and strong generalization. However, this paradigm is hindered by representation misalignment: LLMs process high-level semantic tokens, whereas 3D point clouds convey only dense geometric structures. In prior methods, misalignment limits both input and output. At the input stage, dense point patches require heavy pre-alignment, weakening object-level semantics and confusing similar distractors. At the output stage, predictions depend only on dense features without explicit geometric cues, leading to a loss of fine-grained accuracy. To address these limitations, we present the Point Linguist Model (PLM), a general framework that bridges the representation gap between LLMs and dense 3D point clouds without requiring large-scale pre-alignment between 3D-text or 3D-images. Specifically, we introduce Object-centric Discriminative Representation (OcDR), which learns object-centric tokens that capture target semantics and scene relations under a hard negative-aware training objective. This mitigates the misalignment between LLM tokens and 3D points, enhances resilience to distractors, and facilitates semantic-level reasoning within LLMs. For accurate segmentation, we introduce the Geometric Reactivation Decoder (GRD), which predicts masks by combining OcDR tokens carrying LLM-inferred geometry with corresponding dense features, preserving comprehensive dense features throughout the pipeline. Extensive experiments show that PLM achieves significant improvements of +7.3 mIoU on ScanNetv2 and +6.0 mIoU on Multi3DRefer for 3D referring segmentation, with consistent gains across 7 benchmarks spanning 4 different tasks, demonstrating the effectiveness of comprehensive object-centric reasoning for robust 3D understanding.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 61 of its 73 references that are in the map. First posted on arXiv 9 Sep 2025.
Builds on
61 of 73 references.
References · 61
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2023Segment Anythingcited by 8,234
- 2023Visual Instruction Tuningcited by 7,869
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
- 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
- 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,368
- 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
- 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
38 more
- 2022Language-driven Semantic Segmentationcited by 717
- 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
- 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
- 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
- 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
- 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 321
- 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
- 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 303
- 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
- 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
- 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 237
- 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 187
- 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
- 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
- 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
- 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
- 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
- 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
- 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 124
- 2021Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networkscited by 99
- 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 99
- 2023OneFormer3D: One Transformer for Unified Point Cloud Segmentationcited by 95
- 2023RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understandingcited by 94
- 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
- 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 71
- 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 68
- 2021Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Cloudscited by 55
- 2024LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoningcited by 52
- 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 49
- 2023Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understandingcited by 38
- 20233D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentationcited by 36
- 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 35
- 2025GLUS: Global-Local Reasoning Unified into A Single Large Language Model for Video Segmentationcited by 34
- 2024Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentationcited by 30
- 2024Reason3D: Searching and Reasoning 3D Segmentation via Large Language Modelcited by 28
- 2025The Devil is in Temporal Token: High Quality Video Reasoning Segmentationcited by 26
- 2024Uni3D-LLM: Unifying Point Cloud Perception, Generation and Editing with Large Language Modelscited by 25
- 2024RefMask3D: Language-Guided Transformer for 3D Referring Segmentationcited by 25
50 most cited of 61 All 61 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 20263D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models
- 2026PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Models
- 2024More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding
- 2025Pts3D-LLM: Studying the Impact of Token Structure for 3D Scene Understanding With Large Language Models
- 2025All in One: Visual-Description-Guided Unified Point Cloud Segmentation
- 2026PatchAlign3D: Local Feature Alignment for Dense 3D Shape Understanding
- 2026Training-Free Open-Vocabulary 3D Point-Cloud Segmentation on the Generalized Few-Shot Benchmark
- 2025PGOV3D: Open-Vocabulary 3D Semantic Segmentation with Partial-to-Global Curriculum