mlsanityData to 7 Oct 2026

EgoLM: Multi-Modal Language Model of Egocentric Motions

arXiv 2409.18127 ↗First posted 26 Sep 2024cs.CVcited by 28

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

As the prevalence of wearable devices, learning egocentric motions becomes essential to develop contextual AI. In this work, we present EgoLM, a versatile framework that tracks and understands egocentric motions from multi-modal inputs, e.g., egocentric videos and motion sensors. EgoLM exploits rich contexts for the disambiguation of egomotion tracking and understanding, which are ill-posed under single modality conditions. To facilitate the versatile and multi-modal framework, our key insight is to model the joint distribution of egocentric motions and natural languages using large language models (LLM). Multi-modal sensor inputs are encoded and projected to the joint latent space of language models, and used to prompt motion generation or text generation for egomotion tracking or understanding, respectively. Extensive experiments on large-scale multi-modal human motion dataset validate the effectiveness of EgoLM as a generalist model for universal egocentric learning.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 89 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

61 of 89 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
38 more
  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  4. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  5. 2022Human Motion Diffusion Modelcited by 1,062
  6. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  7. 2017End-to-end Recovery of Human Shape and Posecited by 883
  8. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  9. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  10. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  11. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  12. 2020Jukebox: A Generative Model for Musiccited by 535
  13. 2020Rescaling Egocentric Visioncited by 529
  14. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  15. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  16. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
  17. 2017A simple yet effective baseline for 3d human pose estimationcited by 401
  18. 20183D human pose estimation in video with temporal convolutions and semi-supervised trainingcited by 401
  19. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 357
  20. 2013DeepPose: Human Pose Estimation via Deep Neural Networkscited by 305
  21. 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Modelcited by 282
  22. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  23. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 221
  24. 2023Project Aria: A New Tool for Egocentric Multi-Modal AI Researchcited by 201
  25. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  26. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  27. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 109
  28. 2022AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensingcited by 107
  29. 2023Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Modelcited by 104
  30. 2022Ego-Body Pose Estimation via Ego-Head Pose Estimationcited by 93
  31. 2023An Outlook into the Future of Egocentric Visioncited by 84
  32. 2022PoseGPT: Quantization-based 3D Human Motion Generation and Forecastingcited by 78
  33. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  34. 20226D Rotation Representation For Unconstrained Head Pose Estimationcited by 63
  35. 2023AvatarGPT: All-in-One Framework for Motion Understanding, Planning, Generation and Beyondcited by 63
  36. 2023IMUPoser: Full-Body Pose Estimation using IMUs in Phones, Watches, and Earbudscited by 52
  37. 2023EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensorscited by 48
  38. 2023BoDiffusion: Diffusing Sparse Observations for Full-Body Human Motion Synthesiscited by 44

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 10; 2026: 15.

28 papers.

Sort:

By area:

  1. 2024Estimating Body and Hand Motion in an Ego-sensed Worldcited by 56
  2. 2025Whole-Body Conditioned Egocentric Video Predictioncited by 30
  3. 2024SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characterscited by 18
  4. 2025Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Inputcited by 17
  5. 2025UniEgoMotion: A Unified Model for Egocentric Motion Reconstruction, Forecasting, and Generationcited by 17
  6. 2024Leveraging the Power of MLLMs for Gloss-Free Sign Language Translationcited by 13
  7. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 13
  8. 2025Towards Online Multi-Modal Social Interaction Understandingcited by 12
  9. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  10. 2025ECHO: Ego-Centric modeling of Human-Object interactionscited by 4
  11. 2025IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generationcited by 4
  12. 2026Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsingcited by 4
16 more
  1. 2026ActionPlan: Future-Aware Streaming Motion Synthesis via Frame-Level Action Planningcited by 4
  2. 2025HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenescited by 2
  3. 2025MoLingo: Motion-Language Alignment for Text-to-Human Motion Generationcited by 2
  4. 2026U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generationcited by 2
  5. 2026EgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VRcited by 2
  6. 2026NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Datacited by 2
  7. 2026Leveraging Gaze and Set-of-Mark in VLLMs for Human-Object Interaction Anticipation from Egocentric Videoscited by 2
  8. 2026CLUTCH: Contextualized Language model for Unlocking Text-Conditioned Hand motion modelling in the wildcited by 1
  9. 2026Egocentric Visibility-Aware Human Pose Estimationcited by 1
  10. 2026Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUscited by 1
  11. 2026AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wildcited by 1
  12. 2026GeoMotionGPT: Geometry-Aligned Motion Understanding with Large Language Modelscited by 0
  13. 2026Unified Number-Free Text-to-Motion Generation Via Flow Matchingcited by 0
  14. 2026EgoForce: Robust Online Egocentric Motion Reconstruction via Diffusion Forcingcited by 0
  15. 2026Ego-Human Motion Prediction with 3D-Aware LLMcited by 0
  16. 2026EgoExoMoCap: Distributed Ego-Exo Human Motion Capturecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI
  2. 2026EgoMotion: Hierarchical Reasoning and Diffusion for Egocentric Vision-Language Motion Generation
  3. 2025From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision
  4. 2025PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligence
  5. 2025EgoVLM: Policy Optimization for Egocentric Video Understanding
  6. 2025Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos
  7. 2024Simultaneous Localization and Affordance Prediction of Tasks from Egocentric Video
  8. 2026ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining