# A Unified Framework for Motion Reasoning and Generation in Human Interaction

arXiv: https://arxiv.org/abs/2410.05628 · HTML: https://mlsanity.com/paper/2410.05628/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 13 of its 47 references that are in the map.
- First posted on arXiv 8 Oct 2024.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 13 of 13)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation](https://mlsanity.com/paper/2404.14396/) (2024)
- [MotionLLM: Understanding Human Behaviors from Human Motions and Videos](https://mlsanity.com/paper/2405.20340/) (2024)
- [Large Motion Model for Unified Multi-Modal Motion Generation](https://mlsanity.com/paper/2404.01284/) (2024)
- [ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactions](https://mlsanity.com/paper/2311.17057/) (2023)
- [LLaSM: Large Language and Speech Model](https://mlsanity.com/paper/2308.15930/) (2023)
- [Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs](https://mlsanity.com/paper/2405.17013/) (2024)
- [M3GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation](https://mlsanity.com/paper/2405.16273/) (2024)
- [MotionChain: Conversational Motion Controllers via Multimodal Prompts](https://mlsanity.com/paper/2404.01700/) (2024)
- [in2IN: Leveraging individual Information to Generate Human INteractions](https://mlsanity.com/paper/2404.09988/) (2024)
- [CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2406.10462/) (2024)

## Built on by (most cited first, 5 of 5)

- [MotionGPT3: Human Motion as a Second Modality](https://mlsanity.com/paper/2506.24086/) (2025)
- [PolySLGen: Online Multimodal Speaking-Listening Reaction Generation in Polyadic Interaction](https://mlsanity.com/paper/2604.08125/) (2026)
- [Social Structure Matters in 3D Human-Human Interaction Generation](https://mlsanity.com/paper/2606.24255/) (2026)
- [GeoMotionGPT: Geometry-Aligned Motion Understanding with Large Language Models](https://mlsanity.com/paper/2601.07632/) (2026)
- [MotionMERGE: A Multi-granular Framework for Human Motion Editing, Reasoning, Generation, and Explanation](https://mlsanity.com/paper/2605.18956/) (2026)

## Abstract (arXiv)

Recent advancements in large language models (LLMs) have significantly improved their ability to generate natural and contextually relevant text, enabling more human-like AI interactions. However, generating and understanding interactive human-like motion, where multiple individuals engage in coordinated movements, remains challenging due to the complexity of modeling these interactions. Additionally, a unified and versatile model is needed to handle diverse interactive scenarios, such as chat systems that dynamically adapt to user instructions and assigned roles. To address these challenges, we introduce VIM, the Versatile Interactive Motion-language model, which integrates both language and motion modalities to effectively understand, generate, and control interactive motions in multi-turn conversational contexts. Unlike previous studies that primarily focus on uni-directional tasks such as text-to-motion or motion-to-text, VIM employs a unified architecture capable of simultaneously understanding and generating both motion and text modalities. Given the absence of an appropriate dataset to support this task, we introduce Inter-MT2, a large-scale instruction-tuning dataset containing 82.7K multi-turn interactive motion instructions, covering 153K interactive motion samples. Inter-MT2 spans diverse instructional scenarios, including motion editing, question answering, and story generation, leveraging off-the-shelf large language models and motion diffusion models to construct a broad set of interactive motion instructions. We extensively evaluate the versatility of VIM across multiple interactive motion-related tasks, including motion-to-text, text-to-motion, reaction generation, motion editing, and reasoning about motion sequences.