# MoRAG -- Multi-Fusion Retrieval Augmented Generation for Human Motion

arXiv: https://arxiv.org/abs/2409.12140 · HTML: https://mlsanity.com/paper/2409.12140/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 32 of its 40 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)
- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model](https://mlsanity.com/paper/2304.01116/) (2023)
- [BABEL: Bodies, Action and Behavior with English Labels](https://mlsanity.com/paper/2106.09696/) (2021)
- [Language2Pose: Natural Language Grounded Pose Forecasting](https://mlsanity.com/paper/1907.01108/) (2019)
- [Re-Imagen: Retrieval-Augmented Text-to-Image Generator](https://mlsanity.com/paper/2209.14491/) (2022)
- [TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis](https://mlsanity.com/paper/2305.00976/) (2023)
- [Synthesis of Compositional Animations from Textual Descriptions](https://mlsanity.com/paper/2103.14675/) (2021)
- [TEACH: Temporal Action Composition for 3D Humans](https://mlsanity.com/paper/2209.04066/) (2022)
- [Text2Action: Generative Adversarial Synthesis from Language to Action](https://mlsanity.com/paper/1710.05298/) (2017)
- [FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editing](https://mlsanity.com/paper/2312.15004/) (2023)
- [SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generation](https://mlsanity.com/paper/2304.10417/) (2023)
- [Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrieval](https://mlsanity.com/paper/2308.07648/) (2023)
- [Action-GPT: Leveraging Large-scale Language Models for Improved and Generalized Action Generation](https://mlsanity.com/paper/2211.15603/) (2022)
- [A Cross-Dataset Study for Text-based 3D Human Motion Retrieval](https://mlsanity.com/paper/2405.16909/) (2024)
- [Tri-Modal Motion Retrieval by Learning a Joint Embedding Space](https://mlsanity.com/paper/2403.00691/) (2024)

## Built on by (most cited first, 11 of 11)

- [ReMoMask: Retrieval-Augmented Masked Motion Generation](https://mlsanity.com/paper/2508.02605/) (2025)
- [Unified Multi-Modal Interactive & Reactive 3D Motion Generation via Rectified Flow](https://mlsanity.com/paper/2509.24099/) (2025)
- [Retrievals Can Be Detrimental: Unveiling the Backdoor Vulnerability of Retrieval-Augmented Diffusion Models](https://mlsanity.com/paper/2501.13340/) (2025)
- [Beyond Global Alignment: Fine-Grained Motion-Language Retrieval via Pyramidal Shapley-Taylor Learning](https://mlsanity.com/paper/2601.21904/) (2026)
- [MotionRAG-Diff: A Retrieval-Augmented Diffusion Framework for Long-Term Music-to-Dance Generation](https://mlsanity.com/paper/2506.02661/) (2025)
- [Generating Fit Check Videos with a Handheld Camera](https://mlsanity.com/paper/2505.23886/) (2025)
- [FashionPose: Unified Text-Driven Fashion Synthesis with Joint Geometric and Photometric Control](https://mlsanity.com/paper/2507.13311/) (2025)
- [A Survey of Body and Face Motion: Datasets, Performance Evaluation Metrics and Generative Techniques](https://mlsanity.com/paper/2512.09005/) (2025)
- [ReMoMask-2: Latent Retrieval-Augmented Masked Motion Generation](https://mlsanity.com/paper/2609.08365/) (2026)
- [HUMAN-TCI: Hierarchical Multi-Stream Motion-Aware Network with Torso-Centered Interaction for Text-to-Motion Retrieval](https://mlsanity.com/paper/2609.34430/) (2026)
- [MindWorldBench: Evaluating Mental-State-to-Behavior Reasoning in Image-to-Video Generation](https://mlsanity.com/paper/2609.39147/) (2026)

## Abstract (arXiv)

We introduce MoRAG, a novel multi-part fusion based retrieval-augmented generation strategy for text-based human motion generation. The method enhances motion diffusion models by leveraging additional knowledge obtained through an improved motion retrieval process. By effectively prompting large language models (LLMs), we address spelling errors and rephrasing issues in motion retrieval. Our approach utilizes a multi-part retrieval strategy to improve the generalizability of motion retrieval across the language space. We create diverse samples through the spatial composition of the retrieved motions. Furthermore, by utilizing low-level, part-specific motion information, we can construct motion samples for unseen text descriptions. Our experiments demonstrate that our framework can serve as a plug-and-play module, improving the performance of motion diffusion models. Code, pretrained models and sample videos are available at: https://motion-rag.github.io/