# MoTe: Learning Motion-Text Diffusion Model for Multiple Generation Tasks

arXiv: https://arxiv.org/abs/2411.19786 · HTML: https://mlsanity.com/paper/2411.19786/

## Key facts

- Cited by 14 papers in the map (data through 15 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 7 of its 44 references that are in the map.
- First posted on arXiv 29 Nov 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 7 of 7)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://mlsanity.com/paper/2208.10442/) (2022)
- [SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation](https://mlsanity.com/paper/2309.17448/) (2023)

## Built on by (most cited first, 14 of 14)

- [MotionGPT3: Human Motion as a Second Modality](https://mlsanity.com/paper/2506.24086/) (2025)
- [LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens](https://mlsanity.com/paper/2602.12370/) (2026)
- [ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body](https://mlsanity.com/paper/2512.14234/) (2025)
- [IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation](https://mlsanity.com/paper/2512.10730/) (2025)
- [FG-OrIU: Towards Better Forgetting via Feature-Gradient Orthogonality for Incremental Unlearning](https://mlsanity.com/paper/2601.13578/) (2026)
- [Bridging Semantic and Kinematic Conditions with Diffusion-based Discrete Motion Tokenizer](https://mlsanity.com/paper/2603.19227/) (2026)
- [UniMo: Unified Motion Generation and Understanding with Chain of Thought](https://mlsanity.com/paper/2601.12126/) (2026)
- [FingerCap: Fine-grained Finger-level Hand Motion Captioning](https://mlsanity.com/paper/2511.16951/) (2025)
- [AI-Generated Fall Data: Assessing LLMs and Diffusion Model for Wearable Fall Detection](https://mlsanity.com/paper/2505.04660/) (2025)
- [GeoMotionGPT: Geometry-Aligned Motion Understanding with Large Language Models](https://mlsanity.com/paper/2601.07632/) (2026)
- [Encoder-Free Human Motion Understanding via Structured Motion Descriptions](https://mlsanity.com/paper/2604.21668/) (2026)
- [MUGEN: A Unified Framework for Efficient Motion Understanding and Generation](https://mlsanity.com/paper/2607.27581/) (2026)
- [Open-UniMo: Towards Unified Motion-Language Understanding and Generation in the Open World](https://mlsanity.com/paper/2609.14615/) (2026)
- [BiMoGen: Bidirectional Motion-Text Generation via Unified Masked Discrete Diffusion](https://mlsanity.com/paper/2609.35407/) (2026)

## Abstract (arXiv)

Recently, human motion analysis has experienced great improvement due to inspiring generative models such as the denoising diffusion model and large language model. While the existing approaches mainly focus on generating motions with textual descriptions and overlook the reciprocal task. In this paper, we present~MoTe, a unified multi-modal model that could handle diverse tasks by learning the marginal, conditional, and joint distributions of motion and text simultaneously. MoTe enables us to handle the paired text-motion generation, motion captioning, and text-driven motion generation by simply modifying the input context. Specifically, MoTe is composed of three components: Motion Encoder-Decoder (MED), Text Encoder-Decoder (TED), and Moti-on-Text Diffusion Model (MTDM). In particular, MED and TED are trained for extracting latent embeddings, and subsequently reconstructing the motion sequences and textual descriptions from the extracted embeddings, respectively. MTDM, on the other hand, performs an iterative denoising process on the input context to handle diverse tasks. Experimental results on the benchmark datasets demonstrate the superior performance of our proposed method on text-to-motion generation and competitive performance on motion captioning.