# LLM-Guided Material Inference for 3D Point Clouds

arXiv: https://arxiv.org/abs/2512.03237 · HTML: https://mlsanity.com/paper/2512.03237/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 38 references that are in the map.
- First posted on arXiv 2 Dec 2025.

## Builds on (most cited first, 29 of 29)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies](https://mlsanity.com/paper/2206.04670/) (2022)
- [PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers](https://mlsanity.com/paper/2108.08839/) (2021)
- [ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding](https://mlsanity.com/paper/2212.05171/) (2022)
- [CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation](https://mlsanity.com/paper/2110.02624/) (2021)
- [OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding](https://mlsanity.com/paper/2305.10764/) (2023)
- [Learning to Predict Indoor Illumination from a Single Image](https://mlsanity.com/paper/1704.00090/) (2017)
- [LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent](https://mlsanity.com/paper/2309.12311/) (2023)
- [Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes](https://mlsanity.com/paper/2308.08769/) (2023)
- [Single-Image SVBRDF Capture with a Rendering-Aware Deep Network](https://mlsanity.com/paper/1810.09718/) (2018)
- [Neural Kernel Surface Reconstruction](https://mlsanity.com/paper/2305.19590/) (2023)
- [ParSeNet: A Parametric Surface Fitting Network for 3D Point Clouds](https://mlsanity.com/paper/2003.12181/) (2020)
- [OneFormer3D: One Transformer for Unified Point Cloud Segmentation](https://mlsanity.com/paper/2311.14405/) (2023)
- [Supervised Fitting of Geometric Primitives to 3D Point Clouds](https://mlsanity.com/paper/1811.08988/) (2018)
- [Material Recognition in the Wild with the Materials in Context Database](https://mlsanity.com/paper/1412.0623/) (2014)
- [JoinABLe: Learning Bottom-up Assembly of Parametric CAD Joints](https://mlsanity.com/paper/2111.12772/) (2021)
- [CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generation](https://mlsanity.com/paper/2505.04481/) (2025)
- [HoLa: B-Rep Generation using a Holistic Latent Representation](https://mlsanity.com/paper/2504.14257/) (2025)
- [Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials](https://mlsanity.com/paper/2404.16829/) (2024)
- [Recognizing Material Properties from Images](https://mlsanity.com/paper/1801.03127/) (2018)
- [Better STEP, a format and dataset for boundary representation](https://mlsanity.com/paper/2506.05417/) (2025)
- [PS-CAD: Local Geometry Guidance via Prompting and Selection for CAD Reconstruction](https://mlsanity.com/paper/2405.15188/) (2024)

## Abstract (arXiv)

Most existing 3D shape datasets and models focus solely on geometry, overlooking the material properties that determine how objects appear. We introduce a two-stage large language model (LLM) based method for inferring material composition directly from 3D point clouds with coarse segmentations. Our key insight is to decouple reasoning about what an object is from what it is made of. In the first stage, an LLM predicts the object's semantic; in the second stage, it assigns plausible materials to each geometric segment, conditioned on the inferred semantics. Both stages operate in a zero-shot manner, without task-specific training. Because existing datasets lack reliable material annotations, we evaluate our method using an LLM-as-a-Judge implemented in DeepEval. Across 1,000 shapes from Fusion/ABS and ShapeNet, our method achieves high semantic and material plausibility. These results demonstrate that language models can serve as general-purpose priors for bridging geometric reasoning and material understanding in 3D data.