# LISA-3D: Lifting Language-Image Segmentation to 3D via Multi-View Consistency

arXiv: https://arxiv.org/abs/2512.01008 · HTML: https://mlsanity.com/paper/2512.01008/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 30 Nov 2025.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud](https://mlsanity.com/paper/2003.01251/) (2020)
- [3D-VLA: A 3D Vision-Language-Action Generative World Model](https://mlsanity.com/paper/2403.09631/) (2024)
- [OpenMask3D: Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2306.13631/) (2023)
- [SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation](https://mlsanity.com/paper/2212.04493/) (2022)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos](https://mlsanity.com/paper/2501.04001/) (2025)
- [InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring](https://mlsanity.com/paper/2103.01128/) (2021)
- [SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding](https://mlsanity.com/paper/2401.09340/) (2024)
- [Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance](https://mlsanity.com/paper/2312.10671/) (2023)
- [Anything-3D: Towards Single-view Anything Reconstruction in the Wild](https://mlsanity.com/paper/2304.10261/) (2023)
- [Part123: Part-aware 3D Reconstruction from a Single-view Image](https://mlsanity.com/paper/2405.16888/) (2024)
- [Splat-MOVER: Multi-Stage, Open-Vocabulary Robotic Manipulation via Editable Gaussian Splatting](https://mlsanity.com/paper/2405.04378/) (2024)
- [Search3D: Hierarchical Open-Vocabulary 3D Segmentation](https://mlsanity.com/paper/2409.18431/) (2024)
- [OpenSplat3D: Open-Vocabulary 3D Instance Segmentation using Gaussian Splatting](https://mlsanity.com/paper/2506.07697/) (2025)
- [EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editing](https://mlsanity.com/paper/2410.12836/) (2024)
- [From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs](https://mlsanity.com/paper/2502.20389/) (2025)
- [Ref-SAM3D: Bridging SAM3D with Text for Reference 3D Reconstruction](https://mlsanity.com/paper/2511.19426/) (2025)

## Abstract (arXiv)

Text-driven 3D reconstruction requires masks that understand free-form instructions and remain stable under viewpoint changes. We present LISA-3D, a two-stage framework that adapts the instruction-following segmenter LISA with geometry-aware Low-Rank Adaptation (LoRA) layers while keeping the SAM-3D reconstructor frozen. During training, paired RGB-D frames and camera poses define a differentiable reprojection loss that enforces cross-view agreement without additional 3D-text annotations. At deployment, the adapted segmenter can produce an RGBA prompt for SAM-3D from one RGB image; when registered RGB-D views are available, optional logit fusion further improves the prompt. On ScanRefer and Nr3D, geometry-aware tuning improves both 2D masks and lifted 3D reconstructions while updating only 11.6M parameters. Our results separate geometry-aware training gains from optional multi-view inference gains, providing a modular route from language grounding to object-centric 3D reconstruction.