# CLIP-Actor: Text-Driven Recommendation and Stylization for Animating Human Meshes

arXiv: https://arxiv.org/abs/2206.04382 · HTML: https://mlsanity.com/paper/2206.04382/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 69 references that are in the map.
- First posted on arXiv 9 Jun 2022.

## Builds on (most cited first, 30 of 56)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs](https://mlsanity.com/paper/2112.00724/) (2021)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis](https://mlsanity.com/paper/2104.00677/) (2021)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021)
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021)
- [3D Menagerie: Modeling the 3D shape and pose of animals](https://mlsanity.com/paper/1611.07700/) (2016)
- [BABEL: Bodies, Action and Behavior with English Labels](https://mlsanity.com/paper/2106.09696/) (2021)
- [Language2Pose: Natural Language Grounded Pose Forecasting](https://mlsanity.com/paper/1907.01108/) (2019)
- [Multi-Garment Net: Learning to Dress 3D People from Images](https://mlsanity.com/paper/1908.06903/) (2019)

## Built on by (most cited first, 30 of 41)

- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting](https://mlsanity.com/paper/2311.17061/) (2023)
- [AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Control](https://mlsanity.com/paper/2303.17606/) (2023)
- [Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering](https://mlsanity.com/paper/2312.11360/) (2023)
- [ClipFace: Text-guided Editing of Textured 3D Morphable Models](https://mlsanity.com/paper/2212.01406/) (2022)
- [Understanding Pure CLIP Guidance for Voxel Grid NeRF Models](https://mlsanity.com/paper/2209.15172/) (2022)
- [Being Comes from Not-being: Open-vocabulary Text-to-Motion Generation with Wordless Training](https://mlsanity.com/paper/2210.15929/) (2022)
- [HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances](https://mlsanity.com/paper/2403.01693/) (2024)
- [Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment](https://mlsanity.com/paper/2303.17490/) (2023)
- [PoseFix: Correcting 3D Human Poses with Natural Language](https://mlsanity.com/paper/2309.08480/) (2023)
- [ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model](https://mlsanity.com/paper/2311.17618/) (2023)
- [AvatarBooth: High-Quality and Customizable 3D Human Avatar Generation](https://mlsanity.com/paper/2306.09864/) (2023)
- [ViT-Lens: Towards Omni-modal Representations](https://mlsanity.com/paper/2311.16081/) (2023)
- [Controllable Text-to-3D Generation via Surface-Aligned Gaussian Splatting](https://mlsanity.com/paper/2403.09981/) (2024)
- [What is the Best Automated Metric for Text to Motion Generation?](https://mlsanity.com/paper/2309.10248/) (2023)
- [ZeroAvatar: Zero-shot 3D Avatar Generation from a Single Image](https://mlsanity.com/paper/2305.16411/) (2023)
- [Text2Scene: Text-driven Indoor Scene Stylization with Part-aware Details](https://mlsanity.com/paper/2308.16880/) (2023)
- [XAGen: 3D Expressive Human Avatars Generation](https://mlsanity.com/paper/2311.13574/) (2023)
- [A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigms](https://mlsanity.com/paper/2504.16420/) (2025)
- [TexDreamer: Towards Zero-Shot High-Fidelity 3D Human Texture Generation](https://mlsanity.com/paper/2403.12906/) (2024)
- [PoseScript: Linking 3D Human Poses and Natural Language](https://mlsanity.com/paper/2210.11795/) (2022)
- [STAR: Skeleton-aware Text-based 4D Avatar Generation with In-Network Motion Retargeting](https://mlsanity.com/paper/2406.04629/) (2024)
- [Narrator: Towards Natural Control of Human-Scene Interaction Generation via Relationship Reasoning](https://mlsanity.com/paper/2303.09410/) (2023)
- [TextManiA: Enriching Visual Feature by Text-driven Manifold Augmentation](https://mlsanity.com/paper/2307.14611/) (2023)
- [Text-Guided 3D Face Synthesis -- From Generation to Editing](https://mlsanity.com/paper/2312.00375/) (2023)
- [Semantify: Simplifying the Control of 3D Morphable Models using CLIP](https://mlsanity.com/paper/2308.07415/) (2023)
- [MVControl: Adding Conditional Control to Multi-view Diffusion for Controllable Text-to-3D Generation](https://mlsanity.com/paper/2311.14494/) (2023)
- [PoseEmbroider: Towards a 3D, Visual, Semantic-aware Human Pose Representation](https://mlsanity.com/paper/2409.06535/) (2024)
- [HumanCoser: Layered 3D Human Generation via Semantic-Aware Diffusion Model](https://mlsanity.com/paper/2408.11357/) (2024)

## Abstract (arXiv)

We propose CLIP-Actor, a text-driven motion recommendation and neural mesh stylization system for human mesh animation. CLIP-Actor animates a 3D human mesh to conform to a text prompt by recommending a motion sequence and optimizing mesh style attributes. We build a text-driven human motion recommendation system by leveraging a large-scale human motion dataset with language labels. Given a natural language prompt, CLIP-Actor suggests a text-conforming human motion in a coarse-to-fine manner. Then, our novel zero-shot neural style optimization detailizes and texturizes the recommended mesh sequence to conform to the prompt in a temporally-consistent and pose-agnostic manner. This is distinctive in that prior work fails to generate plausible results when the pose of an artist-designed mesh does not conform to the text from the beginning. We further propose the spatio-temporal view augmentation and mask-weighted embedding attention, which stabilize the optimization process by leveraging multi-frame human motion and rejecting poorly rendered views. We demonstrate that CLIP-Actor produces plausible and human-recognizable style 3D human mesh in motion with detailed geometry and texture solely from a natural language prompt.