mlsanityData to 14 Oct 2026

ChatGarment: Garment Estimation, Generation and Editing via Large Language Models

arXiv 2412.17811 ↗First posted 23 Dec 2024cs.CVcited by 23

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

We introduce ChatGarment, a novel approach that leverages large vision-language models (VLMs) to automate the estimation, generation, and editing of 3D garments from images or text descriptions. Unlike previous methods that struggle in real-world scenarios or lack interactive editing capabilities, ChatGarment can estimate sewing patterns from in-the-wild images or sketches, generate them from text descriptions, and edit garments based on user instructions, all within an interactive dialogue. These sewing patterns can then be draped on a 3D body and animated. This is achieved by finetuning a VLM to directly generate a JSON file that includes both textual descriptions of garment types and styles, as well as continuous numerical attributes. This JSON file is then used to create sewing patterns through a programming parametric model. To support this, we refine the existing programming model, GarmentCode, by expanding its garment type coverage and simplifying its structure for efficient VLM fine-tuning. Additionally, we construct a large-scale dataset of image-to-sewing-pattern and text-to-sewing-pattern pairs through an automated data pipeline. Extensive evaluations demonstrate ChatGarment's ability to accurately reconstruct, generate, and edit garments from multimodal inputs, highlighting its potential to simplify workflows in fashion and gaming applications. Code and data are available at https://chatgarment.github.io/ .

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 65 references that are in the map. First posted on arXiv 23 Dec 2024.

Builds on

56 of 65 references.

References · 56

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  6. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  7. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  8. 2020PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitizationcited by 521
  9. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 460
  10. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 455
  11. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  12. 2021ICON: Implicit Clothed humans Obtained from Normalscited by 283
38 more
  1. 2019Multi-Garment Net: Learning to Dress 3D People from Imagescited by 245
  2. 2023BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motioncited by 236
  3. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 208
  4. 2022ECON: Explicit Clothed humans Optimized via Normal integrationcited by 199
  5. 2019Self-Correction for Human Parsingcited by 180
  6. 2021SMPLicit: Topology-aware Generative Model for Clothed Peoplecited by 155
  7. 2022StyleGAN-Human: A Data-Centric Odyssey of Human Generationcited by 144
  8. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 134
  9. 2020BCNet: Learning Body and Cloth Shape from A Single Imagecited by 120
  10. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 114
  11. 2023TeCH: Text-guided Reconstruction of Lifelike Clothed Humanscited by 104
  12. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 95
  13. 2021The Power of Points for Modeling Humans in Clothingcited by 90
  14. 2023ChatPose: Chatting about 3D Human Posecited by 72
  15. 2024TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Modelscited by 64
  16. 2022ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Modelcited by 62
  17. 20244D-DRESS: A 4D Dataset of Real-world Human Clothing with Semantic Annotationscited by 54
  18. 2018Learning a Shared Shape Space for Multimodal Garment Designcited by 52
  19. 2024DressCode: Autoregressively Sewing and Generating Garments from Text Guidancecited by 44
  20. 2022DrapeNet: Garment Generation and Self-Supervised Drapingcited by 41
  21. 2022Registering Explicit to Implicit: Towards High-Fidelity Garment mesh Reconstruction from Single Imagescited by 39
  22. 2024Can Large Language Models Understand Symbolic Graphics Programs?cited by 35
  23. 2020Codimensional Incremental Potential Contactcited by 34
  24. 2021Generating Datasets of 3D Garments with Sewing Patternscited by 34
  25. 2023Towards Garment Sewing Pattern Reconstruction from a Single Imagecited by 34
  26. 2023ISP: Multi-Layered Garment Draping with Implicit Sewing Patternscited by 32
  27. 2022Neural Point-based Shape Modeling of Humans in Challenging Clothingcited by 29
  28. 2023REC-MV: REconstructing 3D Dynamic Cloth from Monocular Videoscited by 29
  29. 2024GarmentCodeData: A Dataset of 3D Made-to-Measure Garments With Sewing Patternscited by 28
  30. 2023GarmentCode: Programming Parametric Sewing Patternscited by 27
  31. 2024Garment3DGen: 3D Garment Stylization and Texture Generationcited by 26
  32. 2024Uni3D-LLM: Unifying Point Cloud Perception, Generation and Editing with Large Language Modelscited by 25
  33. 2023Ghost on the Shell: An Expressive Representation of General 3D Shapescited by 24
  34. 2023Garment Recovery with Shape and Deformation Priorscited by 24
  35. 2024Re-Thinking Inverse Graphics With Large Language Modelscited by 23
  36. 2024ContourCraft: Learning to Resolve Intersections in Neural Multi-Garment Simulationscited by 23
  37. 2022PERGAMO: Personalized 3D Garments from Monocular Videocited by 22
  38. 2024CloSe: A 3D Clothing Segmentation Dataset and Modelcited by 22

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 14.

23 papers.

Sort:

By area:

  1. 2025Dress-1-to-3: Single Image to Simulation-Ready 3D Outfit with Diffusion Prior and Differentiable Physicscited by 20
  2. 2025Human-Centric Foundation Models: Perception, Generation and Agentic Modelingcited by 12
  3. 2025GarmentDiffusion: 3D Garment Sewing Pattern Generation with Multimodal Diffusion Transformerscited by 9
  4. 2025GarmageNet: A Multimodal Generative Framework for Sewing Pattern Design and Generic Garment Modelingcited by 4
  5. 2025SMooGPT: Stylized Motion Generation using Large Language Modelscited by 4
  6. 2026Image2Garment: Simulation-ready Garment Generation from a Single Imagecited by 3
  7. 2025A Text-to-3D Framework for Joint Generation of CG-Ready Humans and Compatible Garmentscited by 2
  8. 2025D-Garment: Physically Grounded Latent Diffusion for Dynamic Garment Deformationscited by 2
  9. 2026DressWild: Feed-Forward Pose-Agnostic Garment Sewing Pattern Generation from In-the-Wild Imagescited by 2
  10. 2026NGL: Natural Garment Language for Training-Free Sewing Pattern Estimationcited by 2
  11. 2025From Air to Wear: Personalized 3D Digital Fashion with AR/VR Immersive 3D Sketchingcited by 1
  12. 2025Grounding Intelligence in Movementcited by 1
11 more
  1. 2026Learning Sewing Patterns via Latent Flow Matching of Implicit Fieldscited by 1
  2. 2026InverseDraping: Recovering Sewing Patterns from 3D Garment Surfaces via BoxMesh Bridgingcited by 1
  3. 2026AnySurf: Any Surface Generation with Directed Edgecited by 1
  4. 2026Spatio-Temporal Garment Reconstruction Using Diffusion Mapping via Pattern Coordinatescited by 0
  5. 2026OmniFashion: Towards Generalist Fashion Intelligence via Multi-Task Vision-Language Learningcited by 0
  6. 2026FitVTON: Fit-aware Virtual Try-On via Body-Garment Size Controlcited by 0
  7. 2026Fashion-3DLR: A Controllable 3D Garment Generation Using Pairwise Fashion Elements for Intelligent Designcited by 0
  8. 2026TailorCoPilot: Enabling Agentic Pattern Making with Version-Controlled State Trackingcited by 0
  9. 2026GarmentWeaver: Schema-Aware Structured Synthesis for Multimodal Sewing Patternscited by 0
  10. 2026OmniFabric: Coherent UV Space Texture Synthesis for 3D Garment Reconstructioncited by 0
  11. 2026Inverse Garment Design with Visual Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026PatternGSL: A Structured Specification Language for Template-Free and Simulation-Ready 3D Garments
  2. 2025EditGarment: An Instruction-Based Garment Editing Dataset Constructed with Automated MLLM Synthesis and Semantic-Aware Evaluation
  3. 2024AIpparel: A Multimodal Foundation Model for Digital Garments
  4. 2026Garment Particles: A 2D--3D Symmetric Garment Representation for Generation and Editing
  5. 2026SwiftTailor: Efficient 3D Garment Generation with Geometry Image Representation
  6. 2026GarmentSketch: Large-scale Sketch-to-Fashion Benchmark
  7. 2026Dress-ED: Instruction-Guided Editing for Virtual Try-On and Try-Off
  8. 2025Fashion-RAG: Multimodal Fashion Image Editing via Retrieval-Augmented Generation