# ClipMatrix: Text-controlled Creation of 3D Textured Meshes

arXiv: https://arxiv.org/abs/2109.12922 · HTML: https://mlsanity.com/paper/2109.12922/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 7 references that are in the map.
- First posted on arXiv 27 Sep 2021.

## Builds on (most cited first, 5 of 5)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis](https://mlsanity.com/paper/2104.00677/) (2021)
- [Learning to Transfer Texture from Clothing Images to 3D Humans](https://mlsanity.com/paper/2003.02050/) (2020)

## Built on by (most cited first, 30 of 40)

- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](https://mlsanity.com/paper/2212.00774/) (2022)
- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021)
- [CLIP-Mesh: Generating textured meshes from text using pretrained image-text models](https://mlsanity.com/paper/2203.13333/) (2022)
- [Text-To-4D Dynamic Scene Generation](https://mlsanity.com/paper/2301.11280/) (2023)
- [4D-fy: Text-to-4D Generation Using Hybrid Score Distillation Sampling](https://mlsanity.com/paper/2311.17984/) (2023)
- [OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding](https://mlsanity.com/paper/2305.10764/) (2023)
- [VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control](https://mlsanity.com/paper/2407.12781/) (2024)
- [AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers](https://mlsanity.com/paper/2411.18673/) (2024)
- [PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Models](https://mlsanity.com/paper/2212.01558/) (2022)
- [TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decomposition](https://mlsanity.com/paper/2210.11277/) (2022)
- [TC4D: Trajectory-Conditioned Text-to-4D Generation](https://mlsanity.com/paper/2403.17920/) (2024)
- [GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning](https://mlsanity.com/paper/2312.11461/) (2023)
- [ClipFace: Text-guided Editing of Textured 3D Morphable Models](https://mlsanity.com/paper/2212.01406/) (2022)
- [AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars](https://mlsanity.com/paper/2205.08535/) (2022)
- [OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers](https://mlsanity.com/paper/2312.08985/) (2023)
- [Being Comes from Not-being: Open-vocabulary Text-to-Motion Generation with Wordless Training](https://mlsanity.com/paper/2210.15929/) (2022)
- [Lyra 2.0: Explorable Generative 3D Worlds](https://mlsanity.com/paper/2604.13036/) (2026)
- [Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation](https://mlsanity.com/paper/2509.19296/) (2025)
- [ISS: Image as Stepping Stone for Text-Guided 3D Shape Generation](https://mlsanity.com/paper/2209.04145/) (2022)
- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [Surrogate Modeling of Car Drag Coefficient with Depth and Normal Renderings](https://mlsanity.com/paper/2306.06110/) (2023)
- [UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with Authenticity Guided Textures](https://mlsanity.com/paper/2401.11078/) (2024)
- [Text-Driven Stylization of Video Objects](https://mlsanity.com/paper/2206.12396/) (2022)
- [FaceCLIPNeRF: Text-driven 3D Face Manipulation using Deformable Neural Radiance Fields](https://mlsanity.com/paper/2307.11418/) (2023)
- [Towards High-Fidelity Text-Guided 3D Face Generation and Manipulation Using only Images](https://mlsanity.com/paper/2308.16758/) (2023)
- [Multimodal Latent Diffusion Model for Complex Sewing Pattern Generation](https://mlsanity.com/paper/2412.14453/) (2024)
- [SimAvatar: Simulation-Ready Avatars with Layered Hair and Clothing](https://mlsanity.com/paper/2412.09545/) (2024)

## Abstract (arXiv)

If a picture is worth thousand words, a moving 3d shape must be worth a million. We build upon the success of recent generative methods that create images fitting the semantics of a text prompt, and extend it to the controlled generation of 3d objects. We present a novel algorithm for the creation of textured 3d meshes, controlled by text prompts. Our method creates aesthetically pleasing high resolution articulated 3d meshes, and opens new possibilities for automation and AI control of 3d assets. We call it "ClipMatrix" because it leverages CLIP text embeddings to breed new digital 3d creatures, a nod to the Latin meaning of the word "matrix" - "mother". See the online gallery for a full impression of our method's capability.