mlsanityData to 7 Oct 2026

MotionLLM: Understanding Human Behaviors from Human Motions and Videos

arXiv 2405.20340 ↗First posted 30 May 2024cs.CVcited by 89

Not on the map. Why?

Abstract (arXiv)

This study delves into the realm of multi-modality (i.e., video and motion modalities) human behavior understanding by leveraging the powerful capabilities of Large Language Models (LLMs). Diverging from recent LLMs designed for video-only or motion-only understanding, we argue that understanding human behavior necessitates joint modeling from both videos and motion sequences (e.g., SMPL sequences) to capture nuanced body part dynamics and semantics effectively. In light of this, we present MotionLLM, a straightforward yet effective framework for human motion understanding, captioning, and reasoning. Specifically, MotionLLM adopts a unified video-motion training strategy that leverages the complementary advantages of existing coarse video-text data and fine-grained motion-text data to glean rich spatial-temporal insights. Furthermore, we collect a substantial dataset, MoVid, comprising diverse videos, motions, captions, and instructions. Additionally, we propose the MoVid-Bench, with carefully manual annotations, for better evaluation of human behavior understanding on video and motion. Extensive experiments show the superiority of MotionLLM in the caption, spatial-temporal comprehension, and reasoning ability.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Builds on 67 of its 91 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

67 of 91 references.

References · 67

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  8. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  9. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  10. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
38 more
  1. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  2. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  3. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  4. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 683
  5. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  6. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  7. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 567
  8. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  9. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  10. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  11. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  12. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  13. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  14. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
  15. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  16. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  17. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 357
  18. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  19. 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
  20. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233
  21. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  22. 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 220
  23. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  24. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  25. 2021Synthesis of Compositional Animations from Textual Descriptionscited by 177
  26. 2024MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Modelcited by 164
  27. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  28. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  29. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  30. 2017Text2Action: Generative Adversarial Synthesis from Language to Actioncited by 106
  31. 2023HumanTOMATO: Text-aligned Whole-body Motion Generationcited by 103
  32. 2022Towards Diverse and Natural Scene-aware 3D Human Motion Synthesiscited by 91
  33. 2023Unified Human-Scene Interaction via Prompted Chain-of-Contactscited by 91
  34. 2023EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generationcited by 88
  35. 2023DreamWaltz: Make a Scene with Complex 3D Animatable Avatarscited by 82
  36. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  37. 2022UDE: A Unified Driving Engine for Human Motion Generationcited by 67
  38. 2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 49; 2026: 27.

89 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  3. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
  4. 2025MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Spacecited by 82
  5. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 58
  6. 2024The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motioncited by 46
  7. 2024SMooDi: Stylized Motion Diffusion Modelcited by 37
  8. 2025MotionGPT3: Human Motion as a Second Modalitycited by 36
  9. 2024Human Motion Instruction Tuningcited by 29
  10. 2024KinMo: Kinematic-aware Human Motion Understanding and Generationcited by 21
  11. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  12. 2024UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editingcited by 18
38 more
  1. 2025Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Inputcited by 17
  2. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 17
  3. 2025HERO: Human Reaction Generation from Videoscited by 15
  4. 2024MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotationscited by 14
  5. 2024Do Language Models Understand Time?cited by 14
  6. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 14
  7. 2025ChatMotion: A Multimodal Multi-Agent for Human Motion Analysiscited by 13
  8. 2024Is this Generated Person Existed in Real-world? Fine-grained Detecting and Calibrating Abnormal Human-bodycited by 12
  9. 2025Motion-example-controlled Co-speech Gesture Generation Leveraging Large Language Modelscited by 10
  10. 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
  11. 2025Being-M0.5: A Real-Time Controllable Vision-Language-Motion Modelcited by 9
  12. 2025Any2Caption:Interpreting Any Condition to Caption for Controllable Video Generationcited by 8
  13. 2025Text-driven Motion Generation: Overview, Challenges and Directionscited by 8
  14. 2025StyleMotif: Multi-Modal Motion Stylization using Style-Content Cross Fusioncited by 7
  15. 2025ActionArt: Advancing Multimodal Large Models for Fine-Grained Human-Centric Video Understandingcited by 7
  16. 2026FrankenMotion: Part-level Human Motion Generation and Compositioncited by 7
  17. 2025HIS-GPT: Towards 3D Human-In-Scene Multimodal Understandingcited by 6
  18. 2025FSBench: A Figure Skating Benchmark for Advancing Artistic Sports Understandingcited by 6
  19. 2024A Unified Framework for Motion Reasoning and Generation in Human Interactioncited by 5
  20. 2024KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehensioncited by 5
  21. 2025MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Modelscited by 5
  22. 2025Mojito: LLM-Aided Motion Instructor with Jitter-Reduced Inertial Tokenscited by 4
  23. 2025HumanMM: Global Human Motion Recovery from Multi-shot Videoscited by 4
  24. 2025HuMoCon: Concept Discovery for Human Motion Understandingcited by 4
  25. 2025VimoRAG: Video-based Retrieval-augmented 3D Motion Generation for Motion Language Modelscited by 4
  26. 2025SMooGPT: Stylized Motion Generation using Large Language Modelscited by 4
  27. 2025HMVLM: Human Motion-Vision-Lanuage Model via MoE LoRAcited by 4
  28. 2026MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialoguescited by 4
  29. 2026Beyond Global Alignment: Fine-Grained Motion-Language Retrieval via Pyramidal Shapley-Taylor Learningcited by 4
  30. 2026UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generationcited by 4
  31. 2024CigTime: Corrective Instruction Generation Through Inverse Motion Editingcited by 3
  32. 2025LLaVAction: evaluating and training multi-modal large language models for action understandingcited by 3
  33. 2025Towards Fine-Grained Human Motion Video Captioningcited by 3
  34. 2026MotionHiFlow: Text-to-motion via hierarchical flow matchingcited by 3
  35. 2024Fleximo: Towards Flexible Text-to-Human Motion Video Generationcited by 2
  36. 2025CASIM: Composite Aware Semantic Injection for Text to Motion Generationcited by 2
  37. 2025HAIC: Improving Human Action Understanding and Generation with Better Captions for Multi-modal Large Language Modelscited by 2
  38. 2025Unlocking Pretrained LLMs for Motion-Related Multimodal Generation: A Fine-Tuning Approach to Unify Diffusion and Next-Token Predictioncited by 2

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MotionBlind: Probing the Illusion of Motion Understanding in Video-LLMs
  2. 2025MotionSight: Boosting Fine-Grained Motion Understanding in Multimodal LLMs
  3. 2025FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understanding
  4. 2023DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusion
  5. 2026FineBench: Benchmarking and Enhancing Vision-Language Models for Fine-grained Human Activity Understanding
  6. 2026LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens
  7. 2026HumanNet: Scaling Human-centric Video Learning to One Million Hours
  8. 2025Know-Show: Benchmarking Video-Language Models on Spatio-Temporal Grounded Reasoning