mlsanityData to 7 Oct 2026

NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Models

arXiv 2606.04773 ↗First posted 3 Jun 2026cs.CV cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Reliable evaluation of human motion understanding is fundamental to advancing embodied AI, robotics, and animation. However, existing benchmarks suffer from coarse semantic granularity, undifferentiated difficulty, limited annotation quality, and pervasive answer ambiguity, leaving them unable to diagnose where current models fail. To bridge this gap, we introduce NextMotionQA, a comprehensive benchmark that leverages vision-language models (VLMs) for semi-automated, expert-verified dataset. NextMotionQA features three complementary tasks: multiple-choice question answering, video captioning, and fine-grained error correction. Each task is systematically structured across three core semantic axes and stratified into three task complexity levels. Our extensive evaluation of twelve representative VLMs uncovers critical capability gaps and weakness that remain invisible under conventional, single-task evaluations. In a complementary direction, recent work has begun using VLMs as judges for text-to-motion evaluation; we ask whether they show the same degradation under harder tasks. We find that VLMs align strongly with expert ratings on coarse criteria (Cohen's kappa=0.70) but break down on fine-grained, part-level judgment (kappa=0.10), validating the paradigm in its strong regime while clarifying its limits.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 48 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

32 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026FineBench: Benchmarking and Enhancing Vision-Language Models for Fine-grained Human Activity Understanding19 May
  2. 2025FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understanding19 Mar
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May

Other references · 29

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  4. 2025Qwen3-VL Technical Reportcited by 2,534
  5. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  8. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  9. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  10. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 567
  11. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  12. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
17 more
  1. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
  2. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  3. 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Modelcited by 282
  4. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  5. 2024MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Modelcited by 164
  6. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  7. 2023FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editingcited by 105
  8. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 89
  9. 2025MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Spacecited by 82
  10. 2024Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregressioncited by 78
  11. 2024CoMo: Controllable Motion Generation through Language Guided Pose Code Editingcited by 66
  12. 2023Motion Question Answering via Modular Motion Programscited by 27
  13. 2025The Quest for Generalizable Motion Generation: Data, Model, and Evaluationcited by 20
  14. 2025MVGBench: Comprehensive Benchmark for Multi-view Generation Modelscited by 9
  15. 2026FrankenMotion: Part-level Human Motion Generation and Compositioncited by 7
  16. 2026ActionPlan: Future-Aware Streaming Motion Synthesis via Frame-Level Action Planningcited by 4
  17. 2026KPM-Bench: A Kinematic Parsing Motion Benchmark for Fine-grained Motion-centric Video Understandingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026HumanMoveVQA: Can Video MLLMs reason about human movement in videos?
  2. 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models
  3. 2026EgoExo-Next:Benchmarking Vision-Language Models on Visual-Option Next-State and Cross-View Reasoning
  4. 2025Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Grounding
  5. 2026Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Engines
  6. 2025Embodied Scene Understanding for Vision Language Models via MetaVQA
  7. 2025HIS-GPT: Towards 3D Human-In-Scene Multimodal Understanding
  8. 2026EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models