# ExCap3D: Expressive 3D Scene Understanding via Object Captioning with Varying Detail

arXiv: https://arxiv.org/abs/2503.17044 · HTML: https://mlsanity.com/paper/2503.17044/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 58 references that are in the map.
- First posted on arXiv 21 Mar 2025.

## Builds on (most cited first, 30 of 46)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [ScanQA: 3D Question Answering for Spatial Scene Understanding](https://mlsanity.com/paper/2112.10482/) (2021)
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022)
- [Scan2Cap: Context-aware Dense Captioning in RGB-D Scans](https://mlsanity.com/paper/2012.02206/) (2020)
- [LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning](https://mlsanity.com/paper/2311.18651/) (2023)
- [3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment](https://mlsanity.com/paper/2308.04352/) (2023)
- [Scalable 3D Captioning with Pretrained Models](https://mlsanity.com/paper/2306.07279/) (2023)
- [LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness](https://mlsanity.com/paper/2409.18125/) (2024)
- [DenseCap: Fully Convolutional Localization Networks for Dense Captioning](https://mlsanity.com/paper/1511.07571/) (2015)
- [Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers](https://mlsanity.com/paper/2312.08168/) (2023)
- [Multi3DRefer: Grounding Text Description to Multiple 3D Objects](https://mlsanity.com/paper/2309.05251/) (2023)
- [Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes](https://mlsanity.com/paper/2308.08769/) (2023)
- [GARField: Group Anything with Radiance Fields](https://mlsanity.com/paper/2401.09419/) (2024)
- [ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles](https://mlsanity.com/paper/2405.14062/) (2024)
- [End-to-End 3D Dense Captioning with Vote2Cap-DETR](https://mlsanity.com/paper/2301.02508/) (2023)
- [X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioning](https://mlsanity.com/paper/2203.00843/) (2022)
- [Unifying 3D Vision-Language Understanding via Promptable Queries](https://mlsanity.com/paper/2405.11442/) (2024)
- [UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding](https://mlsanity.com/paper/2212.00836/) (2022)
- [MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2401.07745/) (2024)

## Built on by (most cited first, 4 of 4)

- [SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes](https://mlsanity.com/paper/2507.07781/) (2025)
- [IL3D: A Large-Scale Indoor Layout Dataset for LLM-Driven 3D Scene Generation](https://mlsanity.com/paper/2510.12095/) (2025)
- [Efficient3D: A Unified Framework for Adaptive and Debiased Token Reduction in 3D MLLMs](https://mlsanity.com/paper/2604.02689/) (2026)
- [Holo-Captioning: Toward the Text Equivalent of 3D Scenes](https://mlsanity.com/paper/2607.02908/) (2026)

## Abstract (arXiv)

Generating text descriptions of objects in 3D indoor scenes is an important building block of embodied understanding. Existing methods do this by describing objects at a single level of detail, which often does not capture fine-grained details such as varying textures, materials, and shapes of the parts of objects. We propose the task of expressive 3D captioning: given an input 3D scene, describe objects at multiple levels of detail: a high-level object description, and a low-level description of the properties of its parts. To produce such captions, we present ExCap3D, an expressive 3D captioning model which takes as input a 3D scan, and for each detected object in the scan, generates a fine-grained collective description of the parts of the object, along with an object-level description conditioned on the part-level description. We design ExCap3D to encourage semantic consistency between the generated text descriptions, as well as textual similarity in the latent space, to further increase the quality of the generated captions. To enable this task, we generated the ExCap3D Dataset by leveraging a visual-language model (VLM) for multi-view captioning. The ExCap3D Dataset contains captions on the ScanNet++ dataset with varying levels of detail, comprising 190k text descriptions of 34k 3D objects in 947 indoor scenes. Our experiments show that the object- and part-level of detail captions generated by ExCap3D are of higher quality than those produced by state-of-the-art methods, with a Cider score improvement of 17% and 124% for object- and part-level details respectively. Our code, dataset and models will be made publicly available.