# IMoRe: Implicit Program-Guided Reasoning for Human Motion Q&A

arXiv: https://arxiv.org/abs/2508.01984 · HTML: https://mlsanity.com/paper/2508.01984/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 41 references that are in the map.
- First posted on arXiv 4 Aug 2025.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering](https://mlsanity.com/paper/2104.06378/) (2021)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Revisiting Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.13586/) (2021)
- [Compositional Attention Networks for Machine Reasoning](https://mlsanity.com/paper/1803.03067/) (2018)
- [Coarse-to-Fine Decoding for Neural Semantic Parsing](https://mlsanity.com/paper/1805.04793/) (2018)
- [FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editing](https://mlsanity.com/paper/2312.15004/) (2023)
- [SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoning](https://mlsanity.com/paper/2104.05832/) (2021)
- [Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning](https://mlsanity.com/paper/2103.16564/) (2021)
- [NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations](https://mlsanity.com/paper/2303.13483/) (2023)
- [Ask Your Neurons: A Neural-based Approach to Answering Questions about Images](https://mlsanity.com/paper/1505.01121/) (2015)
- [SkeleMotion: A New Representation of Skeleton Joint Sequences Based on Motion Information for 3D Action Recognition](https://mlsanity.com/paper/1907.13025/) (2019)
- [Meta Module Network for Compositional Visual Reasoning](https://mlsanity.com/paper/1910.03230/) (2019)
- [JOLO-GCN: Mining Joint-Centered Light-Weight Information for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2011.07787/) (2020)
- [Self-supervised pre-training and contrastive representation learning for multiple-choice video QA](https://mlsanity.com/paper/2009.08043/) (2020)
- [Motion Question Answering via Modular Motion Programs](https://mlsanity.com/paper/2305.08953/) (2023)
- [Learning to Answer Questions From Image Using Convolutional Neural Network](https://mlsanity.com/paper/1506.00333/) (2015)
- [Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patches](https://mlsanity.com/paper/2405.04771/) (2024)
- [Skeleton-Based Action Segmentation with Multi-Stage Spatial-Temporal Graph Convolutional Neural Networks](https://mlsanity.com/paper/2202.01727/) (2022)
- [3D Concept Grounding on Neural Fields](https://mlsanity.com/paper/2207.06403/) (2022)
- [LocATe: End-to-end Localization of Actions in 3D with Transformers](https://mlsanity.com/paper/2203.10719/) (2022)

## Built on by (most cited first, 5 of 5)

- [LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens](https://mlsanity.com/paper/2602.12370/) (2026)
- [H-GRPO: Permutation-Invariant Reinforcement Learning for Grounded Visual Reasoning](https://mlsanity.com/paper/2606.29915/) (2026)
- [Encoder-Free Human Motion Understanding via Structured Motion Descriptions](https://mlsanity.com/paper/2604.21668/) (2026)
- [MotionMERGE: A Multi-granular Framework for Human Motion Editing, Reasoning, Generation, and Explanation](https://mlsanity.com/paper/2605.18956/) (2026)
- [Fine-grained Human Motion Understanding with Language Models](https://mlsanity.com/paper/2606.20888/) (2026)

## Abstract (arXiv)

Existing human motion Q&A methods rely on explicit program execution, where the requirement for manually defined functional modules may limit the scalability and adaptability. To overcome this, we propose an implicit program-guided motion reasoning (IMoRe) framework that unifies reasoning across multiple query types without manually designed modules. Unlike existing implicit reasoning approaches that infer reasoning operations from question words, our model directly conditions on structured program functions, ensuring a more precise execution of reasoning steps. Additionally, we introduce a program-guided reading mechanism, which dynamically selects multi-level motion representations from a pretrained motion Vision Transformer (ViT), capturing both high-level semantics and fine-grained motion cues. The reasoning module iteratively refines memory representations, leveraging structured program functions to extract relevant information for different query types. Our model achieves state-of-the-art performance on Babel-QA and generalizes to a newly constructed motion Q&A dataset based on HuMMan, demonstrating its adaptability across different motion reasoning datasets. Code and dataset are available at: https://github.com/LUNAProject22/IMoRe.