# MIME: Multimodal Interactive Motion Encoder

arXiv: https://arxiv.org/abs/2607.22702 · HTML: https://mlsanity.com/paper/2607.22702/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 45 of its 54 references that are in the map.
- First posted on arXiv 18 Jul 2026.

## Builds on (most cited first, 30 of 45)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016)
- [ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model](https://mlsanity.com/paper/2304.01116/) (2023)
- [Sampling Matters in Deep Embedding Learning](https://mlsanity.com/paper/1706.07567/) (2017)
- [Hierarchical Question-Image Co-Attention for Visual Question Answering](https://mlsanity.com/paper/1606.00061/) (2016)
- [BABEL: Bodies, Action and Behavior with English Labels](https://mlsanity.com/paper/2106.09696/) (2021)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [OmniControl: Control Any Joint at Any Time for Human Motion Generation](https://mlsanity.com/paper/2310.08580/) (2023)
- [InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions](https://mlsanity.com/paper/2304.05684/) (2023)
- [TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis](https://mlsanity.com/paper/2305.00976/) (2023)
- [EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices](https://mlsanity.com/paper/2112.07642/) (2021)
- [Inter-X: Towards Versatile Human-Human Interaction Analysis](https://mlsanity.com/paper/2312.16051/) (2023)
- [Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregression](https://mlsanity.com/paper/2411.16575/) (2024)
- [Generating Human Interaction Motions in Scenes with Text Control](https://mlsanity.com/paper/2404.10685/) (2024)
- [MotionFix: Text-Driven 3D Human Motion Editing](https://mlsanity.com/paper/2408.00712/) (2024)
- [ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactions](https://mlsanity.com/paper/2311.17057/) (2023)
- [LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning](https://mlsanity.com/paper/2410.07093/) (2024)
- [Make-An-Animation: Large-Scale Text-conditional 3D Human Motion Generation](https://mlsanity.com/paper/2305.09662/) (2023)
- [Duolando: Follower GPT with Off-Policy Reinforcement Learning for Dance Accompaniment](https://mlsanity.com/paper/2403.18811/) (2024)

## Abstract (arXiv)

Text-motion representation learning has advanced rapidly, with growing interest in multi person interactions for animation, AR/VR, and embodied AI. These settings require representations that align language with both individual actor dynamics and the relationships between actors. We introduce the Multimodal Interactive Motion Encoder (MIME), which, to our knowledge, represents the first dedicated multimodal encoder designed specifically for two person interactive motion. MIME captures individual and shared structure using stream based co-attention with explicit interaction features and curriculum based contrastive training. On Inter-X text-motion retrieval, MIME consistently outperforms early and late fusion baselines across gallery sizes, achieving a 12.8% relative improvement in text-to-motion R@1 at a 2,000-sample gallery. We further evaluate MIME as a frozen auxiliary prior within TIMotion and InterMask on the unseen InterHuman dataset. MIME improves semantic alignment metrics while maintaining comparable FID in TIMotion. These results show that interaction aware multimodal encoding improves multi person motion retrieval and transfers across datasets to support downstream motion generation.