mlsanityData to 14 Oct 2026

Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUs

arXiv 2604.21926 ↗First posted 23 Apr 2026cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

Understanding human activities and their surrounding environments typically relies on visual perception, yet cameras pose persistent challenges in privacy, safety, energy efficiency, and scalability. We explore an alternative: 4D perception without vision. Its goal is to reconstruct human motion and 3D scene layouts purely from everyday wearable sensors. For this we introduce IMU-to-4D, a framework that repurposes large language models for non-visual spatiotemporal understanding of human-scene dynamics. IMU-to-4D uses data from a few inertial sensors from earbuds, watches, or smartphones and predicts detailed 4D human motion together with coarse scene structure. Experiments across diverse human-scene datasets show that IMU-to-4D yields more coherent and temporally stable results than SoTA cascaded pipelines, suggesting wearable motion sensors alone can support rich 4D understanding.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 106 of its 123 references that are in the map. First posted on arXiv 23 Apr 2026.

Builds on

106 of 123 references.

References · 106

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  7. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  8. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  9. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  10. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
  11. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  12. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
38 more
  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  2. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  3. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  4. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  5. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  6. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  7. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  8. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  9. 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
  10. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  11. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  12. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  13. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  14. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 295
  15. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  16. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 265
  17. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 249
  18. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 239
  19. 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 227
  20. 2023Object Motion Guided Human Motion Synthesiscited by 212
  21. 2023InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactionscited by 206
  22. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  23. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 202
  24. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  25. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  26. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 158
  27. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 156
  28. 2022Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensorscited by 146
  29. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  30. 2021EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devicescited by 139
  31. 2021Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensorscited by 137
  32. 2024Scaling Up Dynamic Human-Scene Interaction Modelingcited by 137
  33. 2020Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenescited by 135
  34. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 124
  35. 2021DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformercited by 111
  36. 2022AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensingcited by 110
  37. 2017Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUscited by 109
  38. 2023Synthesizing Diverse Human Motions in 3D Indoor Scenescited by 109

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026EgoHTR: Egocentric 4D Demonstrations of Human Terrain Traversalcited by 0
  2. 2026Human-Centric Intelligence in the Era of Foundation Models: A Surveycited by 0