# MoChat: Joints-Grouped Spatio-Temporal Grounding LLM for Multi-Turn Motion Comprehension and Description

arXiv: https://arxiv.org/abs/2410.11404 · HTML: https://mlsanity.com/paper/2410.11404/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 39 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 4 of 4)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)

## Built on by (most cited first, 4 of 4)

- [Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2411.16789/) (2024)
- [FrankenMotion: Part-level Human Motion Generation and Composition](https://mlsanity.com/paper/2601.10909/) (2026)
- [Mojito: LLM-Aided Motion Instructor with Jitter-Reduced Inertial Tokens](https://mlsanity.com/paper/2502.16175/) (2025)
- [Encoder-Free Human Motion Understanding via Structured Motion Descriptions](https://mlsanity.com/paper/2604.21668/) (2026)

## Abstract (arXiv)

Despite continuous advancements in deep learning for understanding human motion, existing models often struggle to accurately identify action timing and specific body parts, typically supporting only single-round interaction. Such limitations in capturing fine-grained motion details reduce their effectiveness in motion understanding tasks. In this paper, we propose MoChat, a multimodal large language model capable of spatio-temporal grounding of human motion and understanding multi-turn dialogue context. To achieve these capabilities, we group the spatial information of each skeleton frame based on human anatomical structure and then apply them with Joints-Grouped Skeleton Encoder, whose outputs are combined with LLM embeddings to create spatio-aware and temporal-aware embeddings separately. Additionally, we develop a pipeline for extracting timestamps from skeleton sequences based on textual annotations, and construct multi-turn dialogues for spatially grounding. Finally, various task instructions are generated for jointly training. Experimental results demonstrate that MoChat achieves state-of-the-art performance across multiple metrics in motion understanding tasks, making it as the first model capable of fine-grained spatio-temporal grounding of human motion.