# ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model

arXiv: https://arxiv.org/abs/2304.01116 · HTML: https://mlsanity.com/paper/2304.01116/

## Key facts

- Cited by 287 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 26 of its 30 references that are in the map.
- First posted on arXiv 3 Apr 2023.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)
- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts](https://mlsanity.com/paper/2207.01696/) (2022)
- [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016)
- [Language2Pose: Natural Language Grounded Pose Forecasting](https://mlsanity.com/paper/1907.01108/) (2019)
- [Re-Imagen: Retrieval-Augmented Text-to-Image Generator](https://mlsanity.com/paper/2209.14491/) (2022)
- [Synthesis of Compositional Animations from Textual Descriptions](https://mlsanity.com/paper/2103.14675/) (2021)
- [TEACH: Temporal Action Composition for 3D Humans](https://mlsanity.com/paper/2209.04066/) (2022)
- [KNN-Diffusion: Image Generation via Large-Scale Retrieval](https://mlsanity.com/paper/2204.02849/) (2022)
- [Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agents](https://mlsanity.com/paper/2101.11101/) (2021)
- [Balanced MSE for Imbalanced Visual Regression](https://mlsanity.com/paper/2203.16427/) (2022)

## Built on by (most cited first, 30 of 287)

- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models](https://mlsanity.com/paper/2310.08529/) (2023)
- [InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion](https://mlsanity.com/paper/2308.16905/) (2023)
- [MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Model](https://mlsanity.com/paper/2404.19759/) (2024)
- [SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation](https://mlsanity.com/paper/2309.17448/) (2023)
- [CG-HOI: Contact-Guided 3D Human-Object Interaction Generation](https://mlsanity.com/paper/2311.16097/) (2023)
- [FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editing](https://mlsanity.com/paper/2312.15004/) (2023)
- [Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos](https://mlsanity.com/paper/2507.15597/) (2025)
- [Generative Image Dynamics](https://mlsanity.com/paper/2309.07906/) (2023)
- [HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Models](https://mlsanity.com/paper/2312.06553/) (2023)
- [Diffusion Models for Reinforcement Learning: A Survey](https://mlsanity.com/paper/2311.01223/) (2023)
- [EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generation](https://mlsanity.com/paper/2312.02256/) (2023)
- [Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregression](https://mlsanity.com/paper/2411.16575/) (2024)
- [Go to Zero: Towards Zero-shot Motion Generation with Million-scale Data](https://mlsanity.com/paper/2507.07095/) (2025)
- [CoMo: Controllable Motion Generation through Language Guided Pose Code Editing](https://mlsanity.com/paper/2403.13900/) (2024)
- [Large Motion Model for Unified Multi-Modal Motion Generation](https://mlsanity.com/paper/2404.01284/) (2024)
- [InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction](https://mlsanity.com/paper/2403.19652/) (2024)
- [AvatarGPT: All-in-One Framework for Motion Understanding, Planning, Generation and Beyond](https://mlsanity.com/paper/2311.16468/) (2023)
- [LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning](https://mlsanity.com/paper/2410.07093/) (2024)
- [MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding](https://mlsanity.com/paper/2410.21747/) (2024)
- [VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents](https://mlsanity.com/paper/2504.09795/) (2025)
- [Interactive Character Control with Auto-Regressive Motion Diffusion Models](https://mlsanity.com/paper/2306.00416/) (2023)
- [ParCo: Part-Coordinating Text-to-Motion Synthesis](https://mlsanity.com/paper/2403.18512/) (2024)
- [ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Model](https://mlsanity.com/paper/2412.14559/) (2024)
- [StableMoFusion: Towards Robust and Efficient Diffusion-based Motion Generation Framework](https://mlsanity.com/paper/2405.05691/) (2024)
- [GENMO: A GENeralist Model for Human MOtion](https://mlsanity.com/paper/2505.01425/) (2025)
- [RoHM: Robust Human Motion Reconstruction via Diffusion](https://mlsanity.com/paper/2401.08570/) (2024)
- [Duolando: Follower GPT with Off-Policy Reinforcement Learning for Dance Accompaniment](https://mlsanity.com/paper/2403.18811/) (2024)

## Abstract (arXiv)

3D human motion generation is crucial for creative industry. Recent advances rely on generative models with domain knowledge for text-driven motion generation, leading to substantial progress in capturing common motions. However, the performance on more diverse motions remains unsatisfactory. In this work, we propose ReMoDiffuse, a diffusion-model-based motion generation framework that integrates a retrieval mechanism to refine the denoising process. ReMoDiffuse enhances the generalizability and diversity of text-driven motion generation with three key designs: 1) Hybrid Retrieval finds appropriate references from the database in terms of both semantic and kinematic similarities. 2) Semantic-Modulated Transformer selectively absorbs retrieval knowledge, adapting to the difference between retrieved samples and the target motion sequence. 3) Condition Mixture better utilizes the retrieval database during inference, overcoming the scale sensitivity in classifier-free guidance. Extensive experiments demonstrate that ReMoDiffuse outperforms state-of-the-art methods by balancing both text-motion consistency and motion quality, especially for more diverse motion generation.