# 3D Highlighter: Localizing Regions on 3D Shapes via Text Descriptions

arXiv: https://arxiv.org/abs/2212.11263 · HTML: https://mlsanity.com/paper/2212.11263/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 50 references that are in the map.
- First posted on arXiv 21 Dec 2022.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022)
- [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021)
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021)
- [Text2LIVE: Text-Driven Layered Image and Video Editing](https://mlsanity.com/paper/2204.02491/) (2022)
- [CLIP-Mesh: Generating textured meshes from text using pretrained image-text models](https://mlsanity.com/paper/2203.13333/) (2022)
- [Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer](https://mlsanity.com/paper/1908.01210/) (2019)
- [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021)
- [Thingi10K: A Dataset of 10,000 3D-Printing Models](https://mlsanity.com/paper/1605.04797/) (2016)
- [CvxNet: Learnable Convex Decomposition](https://mlsanity.com/paper/1909.05736/) (2019)
- [DiffusionNet: Discretization Agnostic Learning on Surfaces](https://mlsanity.com/paper/2012.00888/) (2020)
- [Visual Attribute Transfer through Deep Image Analogy](https://mlsanity.com/paper/1705.01088/) (2017)
- [BAE-NET: Branched Autoencoder for Shape Co-Segmentation](https://mlsanity.com/paper/1903.11228/) (2019)
- [SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation](https://mlsanity.com/paper/1612.00606/) (2016)
- [MeshCNN: A Network with an Edge](https://mlsanity.com/paper/1809.05910/) (2018)
- [ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Model](https://mlsanity.com/paper/2207.09446/) (2022)
- [Subdivision-Based Mesh Convolution Networks](https://mlsanity.com/paper/2106.02285/) (2021)
- [PartNet: A Recursive Part Decomposition Network for Fine-grained and Hierarchical Shape Segmentation](https://mlsanity.com/paper/1903.00709/) (2019)
- [Primal-Dual Mesh Convolutional Neural Networks](https://mlsanity.com/paper/2010.12455/) (2020)
- [MeshWalker: Deep Mesh Understanding by Random Walks](https://mlsanity.com/paper/2006.05353/) (2020)
- [3D Concept Grounding on Neural Fields](https://mlsanity.com/paper/2207.06403/) (2022)

## Built on by (most cited first, 30 of 32)

- [SATR: Zero-Shot Semantic Segmentation of 3D Shapes](https://mlsanity.com/paper/2304.04909/) (2023)
- [Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features](https://mlsanity.com/paper/2311.17024/) (2023)
- [Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databases](https://mlsanity.com/paper/2403.09675/) (2024)
- [Segment Any Mesh](https://mlsanity.com/paper/2408.13679/) (2024)
- [3D Paintbrush: Local Stylization of 3D Shapes with Cascaded Score Distillation](https://mlsanity.com/paper/2311.09571/) (2023)
- [Zero-Shot 3D Shape Correspondence](https://mlsanity.com/paper/2306.03253/) (2023)
- [Videoshop: Localized Semantic Video Editing with Noise-Extrapolated Diffusion Inversion](https://mlsanity.com/paper/2403.14617/) (2024)
- [MeshSegmenter: Zero-Shot Mesh Semantic Segmentation via Texture Synthesis](https://mlsanity.com/paper/2407.13675/) (2024)
- [PartSTAD: 2D-to-3D Part Segmentation Task Adaptation](https://mlsanity.com/paper/2401.05906/) (2024)
- [Back to 3D: Few-Shot 3D Keypoint Detection with Back-Projected 2D Features](https://mlsanity.com/paper/2311.18113/) (2023)
- [3D Part Segmentation via Geometric Aggregation of 2D Visual Features](https://mlsanity.com/paper/2412.04247/) (2024)
- [iSeg: Interactive 3D Segmentation via Interactive Attention](https://mlsanity.com/paper/2404.03219/) (2024)
- [Geometry in Style: 3D Stylization via Surface Normal Deformation](https://mlsanity.com/paper/2503.23241/) (2025)
- [Neural Semantic Surface Maps](https://mlsanity.com/paper/2309.04836/) (2023)
- [Style2Fab: Functionality-Aware Segmentation for Fabricating Personalized 3D Models with Generative AI](https://mlsanity.com/paper/2309.06379/) (2023)
- [Beyond Complete Shapes: A Benchmark for Quantitative Evaluation of 3D Shape Surface Matching Algorithms](https://mlsanity.com/paper/2411.03511/) (2024)
- [DAE-Net: Deforming Auto-Encoder for fine-grained shape co-segmentation](https://mlsanity.com/paper/2311.13125/) (2023)
- [ASIA: Adaptive 3D Segmentation using Few Image Annotations](https://mlsanity.com/paper/2509.24288/) (2025)
- [PatchAlign3D: Local Feature Alignment for Dense 3D Shape Understanding](https://mlsanity.com/paper/2601.02457/) (2026)
- [3DStyleGLIP: Part-Tailored Text-Guided 3D Neural Stylization](https://mlsanity.com/paper/2404.02634/) (2024)
- [HaLo-NeRF: Learning Geometry-Guided Semantics for Exploring Unconstrained Photo Collections](https://mlsanity.com/paper/2404.16845/) (2024)
- [Artiverse: A Diverse and Physically Grounded Dataset for Articulated Objects](https://mlsanity.com/paper/2605.24403/) (2026)
- [GenAnalysis: Joint Shape Analysis by Learning Man-Made Shape Generators with Deformation Regularizations](https://mlsanity.com/paper/2503.00807/) (2025)
- [3D PixBrush: Image-Guided Local Texture Synthesis](https://mlsanity.com/paper/2507.03731/) (2025)
- [Symmetry Informative and Agnostic Feature Disentanglement for 3D Shapes](https://mlsanity.com/paper/2601.14804/) (2026)
- [Text2VDM: Text to Vector Displacement Maps for Expressive and Interactive 3D Sculpting](https://mlsanity.com/paper/2502.20045/) (2025)
- [MechStyle: Augmenting Generative AI with Mechanical Simulation to Create Stylized and Structurally Viable 3D Models](https://mlsanity.com/paper/2509.20571/) (2025)
- [Hierarchical Transformers for Unsupervised 3D Shape Abstraction](https://mlsanity.com/paper/2510.27088/) (2025)
- [MME: Mixture of Mesh Experts with Random Walk Transformer Gating](https://mlsanity.com/paper/2603.00828/) (2026)
- [MeshOn: Intersection-Free Mesh-to-Mesh Composition](https://mlsanity.com/paper/2604.08799/) (2026)

## Abstract (arXiv)

We present 3D Highlighter, a technique for localizing semantic regions on a mesh using text as input. A key feature of our system is the ability to interpret "out-of-domain" localizations. Our system demonstrates the ability to reason about where to place non-obviously related concepts on an input 3D shape, such as adding clothing to a bare 3D animal model. Our method contextualizes the text description using a neural field and colors the corresponding region of the shape using a probability-weighted blend. Our neural optimization is guided by a pre-trained CLIP encoder, which bypasses the need for any 3D datasets or 3D annotations. Thus, 3D Highlighter is highly flexible, general, and capable of producing localizations on a myriad of input shapes. Our code is publicly available at https://github.com/threedle/3DHighlighter.