mlsanityData to 14 Oct 2026

Motion Question Answering via Modular Motion Programs

arXiv 2305.08953 ↗First posted 15 May 2023cs.CV cs.AI cs.LGcited by 27

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

In order to build artificial intelligence systems that can perceive and reason with human behavior in the real world, we must first design models that conduct complex spatio-temporal reasoning over motion sequences. Moving towards this goal, we propose the HumanMotionQA task to evaluate complex, multi-step reasoning abilities of models on long-form human motion sequences. We generate a dataset of question-answer pairs that require detecting motor cues in small portions of motion sequences, reasoning temporally about when events occur, and querying specific motion attributes. In addition, we propose NSPose, a neuro-symbolic method for this task that uses symbolic reasoning and a modular design to ground motion through learning motion concepts, attribute neural operators, and temporal relations. We demonstrate the suitability of NSPose for the HumanMotionQA task, outperforming all baseline methods.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 45 references that are in the map. First posted on arXiv 15 May 2023.

Builds on

29 of 45 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  3. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
  4. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,209
  5. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 780
  6. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 686
  7. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 492
  8. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 408
  9. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 372
  10. 2020Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognitioncited by 343
  11. 2021Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognitioncited by 322
  12. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
17 more
  1. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 265
  2. 2021Revisiting Skeleton-based Action Recognitioncited by 257
  3. 2022FLAME: Free-form Language-based Motion Synthesis & Editingcited by 217
  4. 2022TEACH: Temporal Action Composition for 3D Humanscited by 176
  5. 2017A New Representation of Skeleton Sequences for 3D Action Recognitioncited by 126
  6. 2021Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimationcited by 101
  7. 2022Transformer Inertial Poser: Real-time Human Motion Reconstruction from Sparse IMUs with Simultaneous Terrain Generationcited by 76
  8. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  9. 2023NS3D: Neuro-Symbolic Grounding of 3D Objects and Relationscited by 57
  10. 2021Pose Transformers (POTR): Human Motion Prediction with Non-Autoregressive Transformerscited by 46
  11. 2019SkeleMotion: A New Representation of Skeleton Joint Sequences Based on Motion Information for 3D Action Recognitioncited by 43
  12. 2020JOLO-GCN: Mining Joint-Centered Light-Weight Information for Skeleton-Based Action Recognitioncited by 33
  13. 2022Skeleton-Based Action Segmentation with Multi-Stage Spatial-Temporal Graph Convolutional Neural Networkscited by 20
  14. 2017HuGaDB: Human Gait Database for Activity Recognition from Wearable Inertial Sensor Networkscited by 16
  15. 2021Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognitioncited by 13
  16. 20223D Concept Grounding on Neural Fieldscited by 13
  17. 2022LocATe: End-to-end Localization of Actions in 3D with Transformerscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 10; 2026: 7.

27 papers.

Sort:

By area:

  1. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 90
  2. 2024Motion-Agent: A Conversational Framework for Human Motion Generation with LLMscited by 47
  3. 2023Iterative Motion Editing with Natural Languagecited by 40
  4. 2023What's Left? Concept Grounding with Logic-Enhanced Foundation Modelscited by 33
  5. 2024Human Motion Instruction Tuningcited by 29
  6. 2023Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generationcited by 17
  7. 2024Text-controlled Motion Mamba: Text-Instructed Temporal Grounding of Human Motioncited by 16
  8. 2025ChatMotion: A Multimodal Multi-Agent for Human Motion Analysiscited by 14
  9. 2026LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokenscited by 11
  10. 2025Building Intelligent Agents with Neuro-Symbolic Conceptscited by 10
  11. 2024Naturally Supervised 3D Visual Grounding with Language-Regularized Concept Learnerscited by 8
  12. 2025How Can Objects Help Video-Language Understanding?cited by 8
15 more
  1. 2025HIS-GPT: Towards 3D Human-In-Scene Multimodal Understandingcited by 6
  2. 2025A Study on Neuro-Symbolic Artificial Intelligence: Healthcare Perspectivescited by 5
  3. 2025IMoRe: Implicit Program-Guided Reasoning for Human Motion Q&Acited by 5
  4. 2025HuMoCon: Concept Discovery for Human Motion Understandingcited by 4
  5. 2024MLP: Motion Label Prior for Temporal Sentence Localization in Untrimmed 3D Human Motionscited by 3
  6. 2025Neuro Symbolic Knowledge Reasoning for Procedural Video Question Answeringcited by 3
  7. 2023MDSC: Towards Evaluating the Style Consistency Between Music and Dancecited by 1
  8. 2025MobQA: A Benchmark Dataset for Semantic Understanding of Human Mobility Data through Question Answeringcited by 1
  9. 2026Task-Oriented Communication for Human Action Understanding via Edge-Cloud Co-Inferencecited by 1
  10. 2025ViMoNet: A Multimodal Vision-Language Framework for Human Behavior Understanding from Motion and Videocited by 0
  11. 2026Encoder-Free Human Motion Understanding via Structured Motion Descriptionscited by 0
  12. 2026MotionMERGE: A Multi-granular Framework for Human Motion Editing, Reasoning, Generation, and Explanationcited by 0
  13. 2026NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Modelscited by 0
  14. 2026Fine-grained Human Motion Understanding with Language Modelscited by 0
  15. 2026VSMP-IMU: Video-Grounded Semantic Motion Programs for Sensor-Aware Synthetic IMU Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Neural-Symbolic VideoQA: Learning Compositional Spatio-Temporal Reasoning for Real-world Video Question Answering
  2. 2026From Scene Graphs to Answers: Selective Neuro-Symbolic Reasoning for Autonomous Driving
  3. 2025Think before You Simulate: Symbolic Reasoning to Orchestrate Neural Computation for Counterfactual Question Answering
  4. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videos
  5. 2024STAIR: Spatial-Temporal Reasoning with Auditable Intermediate Results for Video Question Answering
  6. 2026Extending Embodied Question Answering from Perception to Decision
  7. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Robotics
  8. 2026LaMoGen: Language to Motion Generation Through LLM-Guided Symbolic Inference