mlsanityData to 7 Oct 2026

InstructHumans: Editing Animated 3D Human Textures with Instructions

arXiv 2404.04037 ↗First posted 5 Apr 2024cs.CV cs.MMcited by 4

Not on the map. Why?

Abstract (arXiv)

We present InstructHumans, a novel framework for instruction-driven {animatable} 3D human texture editing. Existing text-based 3D editing methods often directly apply Score Distillation Sampling (SDS). SDS, designed for generation tasks, cannot account for the defining requirement of editing -- maintaining consistency with the source avatar. This work shows that naively using SDS harms editing, as it may destroy consistency. We propose a modified SDS for Editing (SDS-E) that selectively incorporates subterms of SDS across diffusion timesteps. We further enhance SDS-E with spatial smoothness regularization and gradient-based viewpoint sampling for edits with sharp and high-fidelity detailing. Incorporating SDS-E into a 3D human texture editing framework allows us to outperform existing 3D editing methods. Our avatars faithfully reflect the textual edits while remaining consistent with the original avatars. Project page: https://jyzhu.top/instruct-humans/.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 43 of its 56 references that are in the map. First posted on arXiv 5 Apr 2024.

Builds on

43 of 56 references.

References · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  7. 2022Classifier-Free Diffusion Guidancecited by 5,717
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  11. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  12. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
31 more
  1. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  2. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  3. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  4. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  5. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  6. 2023GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splattingcited by 331
  7. 2022Perception Prioritized Training of Diffusion Modelscited by 301
  8. 2023DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Modelscited by 154
  9. 2023SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3Dcited by 134
  10. 2023Text-to-3D with Classifier Score Distillationcited by 127
  11. 2022NeRF-Art: Text-Driven Neural Radiance Fields Stylizationcited by 126
  12. 2023Delta Denoising Scorecited by 120
  13. 2023TADA! Text to Animatable Digital Avatarscited by 109
  14. 2023DreamHuman: Animatable 3D Avatars from Textcited by 102
  15. 2023Noise-Free Score Distillationcited by 99
  16. 2023HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generationcited by 88
  17. 2023DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generationcited by 81
  18. 2023ZONE: Zero-Shot Instruction-Guided Local Editingcited by 61
  19. 2022ClipFace: Text-guided Editing of Textured 3D Morphable Modelscited by 59
  20. 2023Learning Locally Editable Virtual Humanscited by 54
  21. 2023Instruct 3D-to-3D: Text Instruction Guided 3D-to-3D conversioncited by 47
  22. 2023Text-Guided Generation and Editing of Compositional 3D Avatarscited by 39
  23. 2023SHAP-EDITOR: Instruction-guided Latent 3D Editing in Secondscited by 36
  24. 2023AvatarStudio: Text-driven Editing of 3D Dynamic Human Head Avatarscited by 29
  25. 2023Stable Score Distillation for High-Quality 3D Generationcited by 27
  26. 2023Chupa: Carving 3D Clothed Humans from Skinned Shape Priors using 2D Diffusion Probabilistic Modelscited by 20
  27. 2023Enhancing High-Resolution 3D Generation through Pixel-wise Gradient Clippingcited by 17
  28. 2023FaceCLIPNeRF: Text-driven 3D Face Manipulation using Deformable Neural Radiance Fieldscited by 12
  29. 2023Progressive Text-to-3D Generation for Automatic 3D Prototypingcited by 8
  30. 2023Exploring Sparse Spatial Relation in Graph Inference for Text-Based VQAcited by 6
  31. 2023PaintHuman: Towards High-fidelity Text-to-3D Human Texturing via Denoised Score Distillationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2024DreamCatalyst: Fast and High-Quality 3D Editing via Controlling Editability and Identity Preservationcited by 13
  2. 2026Gaussian Wardrobe: Compositional 3D Gaussian Avatars for Free-Form Virtual Try-Oncited by 2
  3. 2026RelaxFlow: Text-Driven Amodal 3D Generationcited by 1
  4. 2025Mastering Regional 3DGS: Locating, Initializing, and Editing with Diverse 2D Priorscited by 0