mlsanityData to 7 Oct 2026

Iterative Motion Editing with Natural Language

arXiv 2312.11538 ↗First posted 15 Dec 2023cs.GR cs.CVcited by 39

On the map: Diffusion models → Video generation

Abstract (arXiv)

Text-to-motion diffusion models can generate realistic animations from text prompts, but do not support fine-grained motion editing controls. In this paper, we present a method for using natural language to iteratively specify local edits to existing character animations, a task that is common in most computer animation workflows. Our key idea is to represent a space of motion edits using a set of kinematic motion editing operators (MEOs) whose effects on the source motion is well-aligned with user expectations. We provide an algorithm that leverages pre-existing language models to translate textual descriptions of motion edits into source code for programs that define and execute sequences of MEOs on a source animation. We execute MEOs by first translating them into keyframe constraints, and then use diffusion-based motion models to generate output motions that respect these constraints. Through a user study and quantitative evaluation, we demonstrate that our system can perform motion edits that respect the animator's editing intent, remain faithful to the original animation (it edits the original animation, but does not dramatically change it), and yield realistic character animation results.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 45 of its 73 references that are in the map. First posted on arXiv 15 Dec 2023.

Builds on

45 of 73 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  2. 2023Human Motion Diffusion as a Generative Prior2 Mar
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  4. 2022Human Motion Diffusion Model29 Sep
  5. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model31 Aug

Other references · 40

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  11. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  12. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
28 more
  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  2. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  3. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  4. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  5. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  6. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  7. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  8. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  9. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  10. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  11. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  12. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
  13. 2022EDGE: Editable Dance Generation From Musiccited by 284
  14. 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
  15. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  16. 2023Object Motion Guided Human Motion Synthesiscited by 206
  17. 2021Character Controllers Using Motion VAEscited by 182
  18. 2023Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusioncited by 137
  19. 2020Unpaired Motion Style Transfer from Video to Animationcited by 94
  20. 2023CALM: Conditional Adversarial Latent Models for Directable Virtual Characterscited by 91
  21. 2023SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generationcited by 62
  22. 2022Implicit Neural Representations for Variable Length Human Motion Generationcited by 61
  23. 2023PADL: Language-Directed Physics-Based Character Controlcited by 47
  24. 2023PoseFix: Correcting 3D Human Poses with Natural Languagecited by 38
  25. 2023Motion Question Answering via Modular Motion Programscited by 27
  26. 2022Dance Style Transfer with Cross-modal Transformercited by 20
  27. 2021ProtoRes: Proto-Residual Network for Pose Authoring via Learned Inverse Kinematicscited by 17
  28. 2021Hierarchical Motion Understanding via Motion Programscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 18; 2026: 10.

39 papers.

Sort:

By area:

  1. 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 103
  2. 2024CoMo: Controllable Motion Generation through Language Guided Pose Code Editingcited by 66
  3. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  4. 2024MotionFix: Text-Driven 3D Human Motion Editingcited by 63
  5. 2024Motion-Agent: A Conversational Framework for Human Motion Generation with LLMscited by 46
  6. 2024Monkey See, Monkey Do: Harnessing Self-attention in Motion Diffusion for Zero-shot Motion Transfercited by 30
  7. 2025MotionLab: Unified Human Motion Generation and Editing via the Motion-Condition-Motion Paradigmcited by 30
  8. 2024Unimotion: Unifying 3D Human Motion Synthesis and Understandingcited by 29
  9. 2025BlenderGym: Benchmarking Foundational Model Systems for Graphics Editingcited by 27
  10. 2025SimMotionEdit: Text-Based Human Motion Editing with Motion Similarity Predictioncited by 26
  11. 2024Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesiscited by 25
  12. 2025FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placementcited by 24
27 more
  1. 2024BlenderAlchemy: Editing 3D Graphics with Vision-Language Modelscited by 18
  2. 2024Pay Attention and Move Better: Harnessing Attention for Interactive Motion Generation and Training-free Editingcited by 17
  3. 2024Text-controlled Motion Mamba: Text-Instructed Temporal Grounding of Human Motioncited by 16
  4. 2025Sketch2Anim: Towards Transferring Sketch Storyboards into 3D Animationcited by 16
  5. 2025SocialGen: Modeling Multi-Human Social Interaction with Language Modelscited by 14
  6. 2025Dynamic Motion Blending for Versatile Motion Editingcited by 13
  7. 2025MoVer: Motion Verification for Motion Graphics Animationscited by 10
  8. 2025Motion-example-controlled Co-speech Gesture Generation Leveraging Large Language Modelscited by 10
  9. 2025CoBRA: Programming Cognitive Bias in Social Agents Using Classic Social Science Experimentscited by 8
  10. 2025MotionVerse: A Unified Multimodal Framework for Motion Comprehension, Generation and Editingcited by 5
  11. 2025Generative Motion Infilling From Imprecisely Timed Keyframescited by 4
  12. 2026Omni-Supervised Motion Editing: Balancing Change and Invariance through Positive-Negative Learningcited by 4
  13. 2025Spatial-Temporal Multi-Scale Quantization for Flexible Motion Generationcited by 3
  14. 2025CASIM: Composite Aware Semantic Injection for Text to Motion Generationcited by 2
  15. 2026Cross-Axis Feature Fusion with Joint-Wise Motion Difference Prediction for Text-Based 3D Human Motion Editingcited by 2
  16. 2025ClusterStyle: Modeling Intra-Style Diversity with Prototypical Clustering for Stylized Motion Generationcited by 1
  17. 2026InfiniteDance: Scalable 3D Dance Generation Towards in-the-wild Generalizationcited by 1
  18. 2026ExpertEdit: Learning Skill-Aware Motion Editing from Expert Videoscited by 1
  19. 2026Spatial Temporal Synergy: Balancing Change and Invariance in Text Driven 3D Human Motion Editingcited by 1
  20. 2025Generating Detailed Character Motion from Blocking Posescited by 0
  21. 2025Towards Arbitrary Motion Completing via Hierarchical Continuous Representationcited by 0
  22. 2025DeMoGen: Towards Decompositional Human Motion Generation with Energy-Based Diffusion Modelscited by 0
  23. 2026Discovering High Level Patterns from Simulation Tracescited by 0
  24. 2026MotionMERGE: A Multi-granular Framework for Human Motion Editing, Reasoning, Generation, and Explanationcited by 0
  25. 2026DrawMotion: Generating 3D Human Motions by Freehand Drawingcited by 0
  26. 2026Text Dictates, Music Decorates: Energy-based Attention for Editable Dance Motion Generationcited by 0
  27. 2026Text2Sim: Agentic Physics-Based Simulation Generation with Distilled Expertisecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022FLAME: Free-form Language-based Motion Synthesis & Editing
  2. 2023Enhanced Fine-grained Motion Diffusion for Text-driven Human Motion Synthesis
  3. 2024Autonomous Character-Scene Interaction Synthesis from Text Instruction
  4. 2024Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation
  5. 2024FreeMotion: MoCap-Free Human Motion Synthesis with Multimodal Large Language Models
  6. 2025MotionFlux: Efficient Text-Guided Motion Generation through Rectified Flow Matching and Preference Alignment
  7. 2023Motion Flow Matching for Human Motion Synthesis and Editing
  8. 2025MotionV2V: Editing Motion in a Video