mlsanityData to 7 Oct 2026

LISA-3D: Lifting Language-Image Segmentation to 3D via Multi-View Consistency

arXiv 2512.01008 ↗First posted 30 Nov 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Text-driven 3D reconstruction requires masks that understand free-form instructions and remain stable under viewpoint changes. We present LISA-3D, a two-stage framework that adapts the instruction-following segmenter LISA with geometry-aware Low-Rank Adaptation (LoRA) layers while keeping the SAM-3D reconstructor frozen. During training, paired RGB-D frames and camera poses define a differentiable reprojection loss that enforces cross-view agreement without additional 3D-text annotations. At deployment, the adapted segmenter can produce an RGBA prompt for SAM-3D from one RGB image; when registered RGB-D views are available, optional logit fusion further improves the prompt. On ScanRefer and Nr3D, geometry-aware tuning improves both 2D masks and lifted 3D reconstructions while updating only 11.6M parameters. Our results separate geometry-aware training gains from optional multi-view inference gains, providing a modular route from language grounding to object-centric 3D reconstruction.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 30 Nov 2025.

Builds on

27 of 36 references.

References · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Segment Anythingcited by 8,158
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  7. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  8. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  9. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  10. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  11. 2020Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloudcited by 335
  12. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
15 more
  1. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  2. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 287
  3. 2025SAM 3D: 3Dfy Anything in Imagescited by 230
  4. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  5. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 140
  6. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  7. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  8. 2023Anything-3D: Towards Single-view Anything Reconstruction in the Wildcited by 91
  9. 2024Part123: Part-aware 3D Reconstruction from a Single-view Imagecited by 54
  10. 2024Splat-MOVER: Multi-Stage, Open-Vocabulary Robotic Manipulation via Editable Gaussian Splattingcited by 37
  11. 2024Search3D: Hierarchical Open-Vocabulary 3D Segmentationcited by 31
  12. 2025OpenSplat3D: Open-Vocabulary 3D Instance Segmentation using Gaussian Splattingcited by 19
  13. 2024EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editingcited by 14
  14. 2025From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMscited by 7
  15. 2025Ref-SAM3D: Bridging SAM3D with Text for Reference 3D Reconstructioncited by 1

Built on by

Not cited yet.