# Enhancing Motion Variation in Text-to-Motion Models via Pose and Video Conditioned Editing

arXiv: https://arxiv.org/abs/2410.08931 · HTML: https://mlsanity.com/paper/2410.08931/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 19 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Builds on (most cited first, 13 of 13)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [PARE: Part Attention Regressor for 3D Human Body Estimation](https://mlsanity.com/paper/2104.08527/) (2021)
- [FLAME: Free-form Language-based Motion Synthesis & Editing](https://mlsanity.com/paper/2209.00349/) (2022)
- [AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanism](https://mlsanity.com/paper/2309.00796/) (2023)
- [Fg-T2M: Fine-Grained Text-Driven Human Motion Generation via Diffusion Model](https://mlsanity.com/paper/2309.06284/) (2023)
- [Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model](https://mlsanity.com/paper/2310.07222/) (2023)
- [Plan, Posture and Go: Towards Open-World Text-to-Motion Generation](https://mlsanity.com/paper/2312.14828/) (2023)

## Abstract (arXiv)

Text-to-motion models that generate sequences of human poses from textual descriptions are garnering significant attention. However, due to data scarcity, the range of motions these models can produce is still limited. For instance, current text-to-motion models cannot generate a motion of kicking a football with the instep of the foot, since the training data only includes martial arts kicks. We propose a novel method that uses short video clips or images as conditions to modify existing basic motions. In this approach, the model's understanding of a kick serves as the prior, while the video or image of a football kick acts as the posterior, enabling the generation of the desired motion. By incorporating these additional modalities as conditions, our method can create motions not present in the training set, overcoming the limitations of text-motion datasets. A user study with 26 participants demonstrated that our approach produces unseen motions with realism comparable to commonly represented motions in text-motion datasets (e.g., HumanML3D), such as walking, running, squatting, and kicking.