# Iterative Motion Editing with Natural Language

arXiv: https://arxiv.org/abs/2312.11538 · HTML: https://mlsanity.com/paper/2312.11538/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 45 of its 73 references that are in the map.
- First posted on arXiv 15 Dec 2023.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 45)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [Human Motion Diffusion as a Generative Prior](https://mlsanity.com/paper/2303.01418/) (2023)
- [EDGE: Editable Dance Generation From Music](https://mlsanity.com/paper/2211.10658/) (2022)

## Built on by (most cited first, 30 of 40)

- [CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control](https://mlsanity.com/paper/2410.03441/) (2024)
- [CoMo: Controllable Motion Generation through Language Guided Pose Code Editing](https://mlsanity.com/paper/2403.13900/) (2024)
- [Large Motion Model for Unified Multi-Modal Motion Generation](https://mlsanity.com/paper/2404.01284/) (2024)
- [MotionFix: Text-Driven 3D Human Motion Editing](https://mlsanity.com/paper/2408.00712/) (2024)
- [Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs](https://mlsanity.com/paper/2405.17013/) (2024)
- [Monkey See, Monkey Do: Harnessing Self-attention in Motion Diffusion for Zero-shot Motion Transfer](https://mlsanity.com/paper/2406.06508/) (2024)
- [BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing](https://mlsanity.com/paper/2504.01786/) (2025)
- [Unimotion: Unifying 3D Human Motion Synthesis and Understanding](https://mlsanity.com/paper/2409.15904/) (2024)
- [MotionLab: Unified Human Motion Generation and Editing via the Motion-Condition-Motion Paradigm](https://mlsanity.com/paper/2502.02358/) (2025)
- [SimMotionEdit: Text-Based Human Motion Editing with Motion Similarity Prediction](https://mlsanity.com/paper/2503.18211/) (2025)
- [Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesis](https://mlsanity.com/paper/2412.06786/) (2024)
- [FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placement](https://mlsanity.com/paper/2503.04919/) (2025)
- [BlenderAlchemy: Editing 3D Graphics with Vision-Language Models](https://mlsanity.com/paper/2404.17672/) (2024)
- [Pay Attention and Move Better: Harnessing Attention for Interactive Motion Generation and Training-free Editing](https://mlsanity.com/paper/2410.18977/) (2024)
- [Sketch2Anim: Towards Transferring Sketch Storyboards into 3D Animation](https://mlsanity.com/paper/2504.19189/) (2025)
- [Text-controlled Motion Mamba: Text-Instructed Temporal Grounding of Human Motion](https://mlsanity.com/paper/2404.11375/) (2024)
- [Dynamic Motion Blending for Versatile Motion Editing](https://mlsanity.com/paper/2503.20724/) (2025)
- [SocialGen: Modeling Multi-Human Social Interaction with Language Models](https://mlsanity.com/paper/2503.22906/) (2025)
- [Motion-example-controlled Co-speech Gesture Generation Leveraging Large Language Models](https://mlsanity.com/paper/2507.20220/) (2025)
- [MoVer: Motion Verification for Motion Graphics Animations](https://mlsanity.com/paper/2502.13372/) (2025)
- [CoBRA: Programming Cognitive Bias in Social Agents Using Classic Social Science Experiments](https://mlsanity.com/paper/2509.13588/) (2025)
- [MotionVerse: A Unified Multimodal Framework for Motion Comprehension, Generation and Editing](https://mlsanity.com/paper/2509.23635/) (2025)
- [Omni-Supervised Motion Editing: Balancing Change and Invariance through Positive-Negative Learning](https://mlsanity.com/paper/2605.30969/) (2026)
- [Generative Motion Infilling From Imprecisely Timed Keyframes](https://mlsanity.com/paper/2503.01016/) (2025)
- [Spatial-Temporal Multi-Scale Quantization for Flexible Motion Generation](https://mlsanity.com/paper/2508.08991/) (2025)
- [Cross-Axis Feature Fusion with Joint-Wise Motion Difference Prediction for Text-Based 3D Human Motion Editing](https://mlsanity.com/paper/2606.01014/) (2026)
- [CASIM: Composite Aware Semantic Injection for Text to Motion Generation](https://mlsanity.com/paper/2502.02063/) (2025)
- [ClusterStyle: Modeling Intra-Style Diversity with Prototypical Clustering for Stylized Motion Generation](https://mlsanity.com/paper/2512.02453/) (2025)
- [InfiniteDance: Scalable 3D Dance Generation Towards in-the-wild Generalization](https://mlsanity.com/paper/2603.13375/) (2026)
- [ExpertEdit: Learning Skill-Aware Motion Editing from Expert Videos](https://mlsanity.com/paper/2604.10466/) (2026)

## Abstract (arXiv)

Text-to-motion diffusion models can generate realistic animations from text prompts, but do not support fine-grained motion editing controls. In this paper, we present a method for using natural language to iteratively specify local edits to existing character animations, a task that is common in most computer animation workflows. Our key idea is to represent a space of motion edits using a set of kinematic motion editing operators (MEOs) whose effects on the source motion is well-aligned with user expectations. We provide an algorithm that leverages pre-existing language models to translate textual descriptions of motion edits into source code for programs that define and execute sequences of MEOs on a source animation. We execute MEOs by first translating them into keyframe constraints, and then use diffusion-based motion models to generate output motions that respect these constraints. Through a user study and quantitative evaluation, we demonstrate that our system can perform motion edits that respect the animator's editing intent, remain faithful to the original animation (it edits the original animation, but does not dramatically change it), and yield realistic character animation results.