mlsanityData to 7 Oct 2026

Semantify: Simplifying the Control of 3D Morphable Models using CLIP

arXiv 2308.07415 ↗First posted 14 Aug 2023cs.CV cs.GRcited by 7

Not on the map. Why?

Abstract (arXiv)

We present Semantify: a self-supervised method that utilizes the semantic power of CLIP language-vision foundation model to simplify the control of 3D morphable models. Given a parametric model, training data is created by randomly sampling the model's parameters, creating various shapes and rendering them. The similarity between the output images and a set of word descriptors is calculated in CLIP's latent space. Our key idea is first to choose a small set of semantically meaningful and disentangled descriptors that characterize the 3DMM, and then learn a non-linear mapping from scores across this set to the parametric coefficients of the given 3DMM. The non-linear mapping is defined by training a neural network without a human-in-the-loop. We present results on numerous 3DMMs: body shape models, face shape and expression models, as well as animal shapes. We demonstrate how our method defines a simple slider interface for intuitive modeling, and show how the mapping can be used to instantly fit a 3D parametric body shape to in-the-wild images.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 31 of its 47 references that are in the map. First posted on arXiv 14 Aug 2023.

Builds on

31 of 47 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  4. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  5. 2022Human Motion Diffusion Modelcited by 1,062
  6. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  7. 2017End-to-end Recovery of Human Shape and Posecited by 883
  8. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  9. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  10. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  11. 2020Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstructioncited by 465
  12. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
19 more
  1. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  2. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  3. 20163D Menagerie: Modeling the 3D shape and pose of animalscited by 310
  4. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  5. 20193D Morphable Face Models -- Past, Present and Futurecited by 269
  6. 2022EMOCA: Emotion Driven Monocular Face Capture and Animationcited by 254
  7. 2020FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Predictioncited by 196
  8. 2020Monocular Expressive Body Regression through Body-Driven Attentioncited by 155
  9. 2017Morphable Face Models - An Open Frameworkcited by 138
  10. 2021Collaborative Regression of Expressive Bodies using Moderationcited by 137
  11. 2021SPEC: Seeing People in the Wild with an Estimated Cameracited by 134
  12. 2022NeRF-Art: Text-Driven Neural Radiance Fields Stylizationcited by 126
  13. 2022FaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Datasetcited by 98
  14. 2017Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hzcited by 91
  15. 2021CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directionscited by 78
  16. 2022ClipFace: Text-guided Editing of Textured 3D Morphable Modelscited by 59
  17. 2022Accurate 3D Body Shape Regression using Metric and Semantic Attributescited by 49
  18. 2022CLIP-Actor: Text-Driven Recommendation and Stylization for Animating Human Meshescited by 41
  19. 2022SUPR: A Sparse Unified Part-Based Human Representationcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 1.

7 papers.

  1. 2024UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editingcited by 18
  2. 2024PuzzleAvatar: Assembling 3D Avatars from Personal Albumscited by 16
  3. 2024AWOL: Analysis WithOut synthesis using Languagecited by 12
  4. 2025Shape My Moves: Text-Driven Shape-Aware Synthesis of Human Motionscited by 11
  5. 2025UP2You: Fast Reconstruction of Yourself from Unconstrained Photo Collectionscited by 10
  6. 2025Odo: Depth-Guided Diffusion for Identity-Preserving Body Reshapingcited by 0
  7. 2026Odoriko: A Shape-Aware Multimodal Diffusion Framework for Human Motioncited by 0