# Guided Attention for Interpretable Motion Captioning

arXiv: https://arxiv.org/abs/2310.07324 · HTML: https://mlsanity.com/paper/2310.07324/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 11 Oct 2023.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts](https://mlsanity.com/paper/2207.01696/) (2022)
- [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016)
- [Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning](https://mlsanity.com/paper/1612.01887/) (2016)
- [Synthesis of Compositional Animations from Textual Descriptions](https://mlsanity.com/paper/2103.14675/) (2021)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks](https://mlsanity.com/paper/1705.06400/) (2017)
- [Attention Correctness in Neural Image Captioning](https://mlsanity.com/paper/1605.09553/) (2016)
- [Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning](https://mlsanity.com/paper/1706.01231/) (2017)
- [Supervising Neural Attention Models for Video Captioning by Human Gaze Data](https://mlsanity.com/paper/1707.06029/) (2017)

## Built on by (most cited first, 5 of 5)

- [Unlocking Pretrained LLMs for Motion-Related Multimodal Generation: A Fine-Tuning Approach to Unify Diffusion and Next-Token Prediction](https://mlsanity.com/paper/2503.06119/) (2025)
- [MoCLIP: Motion-Aware Fine-Tuning and Distillation of CLIP for Human Motion Generation](https://mlsanity.com/paper/2505.10810/) (2025)
- [Transformer with Controlled Attention for Synchronous Motion Captioning](https://mlsanity.com/paper/2409.09177/) (2024)
- [Hierarchical Motion Captioning Utilizing External Text Data Source](https://mlsanity.com/paper/2509.01471/) (2025)
- [MoRL: Reinforced Reasoning for Unified Motion Understanding and Generation](https://mlsanity.com/paper/2602.14534/) (2026)

## Abstract (arXiv)

Diverse and extensive work has recently been conducted on text-conditioned human motion generation. However, progress in the reverse direction, motion captioning, has seen less comparable advancement. In this paper, we introduce a novel architecture design that enhances text generation quality by emphasizing interpretability through spatio-temporal and adaptive attention mechanisms. To encourage human-like reasoning, we propose methods for guiding attention during training, emphasizing relevant skeleton areas over time and distinguishing motion-related words. We discuss and quantify our model's interpretability using relevant histograms and density distributions. Furthermore, we leverage interpretability to derive fine-grained information about human motion, including action localization, body part identification, and the distinction of motion-related words. Finally, we discuss the transferability of our approaches to other tasks. Our experiments demonstrate that attention guidance leads to interpretable captioning while enhancing performance compared to higher parameter-count, non-interpretable state-of-the-art systems. The code is available at: https://github.com/rd20karim/M2T-Interpretable.