mlsanityData to 7 Oct 2026

Humans in 4D: Reconstructing and Tracking Humans with Transformers

arXiv 2305.20091 ↗First posted 31 May 2023cs.CVcited by 344

Not on the map. Why?

Cited by 31 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present an approach to reconstruct humans and track them over time. At the core of our approach, we propose a fully "transformerized" version of a network for human mesh recovery. This network, HMR 2.0, advances the state of the art and shows the capability to analyze unusual poses that have in the past been difficult to reconstruct from single images. To analyze video, we use 3D reconstructions from HMR 2.0 as input to a tracking system that operates in 3D. This enables us to deal with multiple people and maintain identities through occlusion events. Our complete approach, 4DHumans, achieves state-of-the-art results for tracking people from monocular video. Furthermore, we demonstrate the effectiveness of HMR 2.0 on the downstream task of action recognition, achieving significant improvements over previous pose-based action recognition approaches. Our code and models are available on the project website: https://shubham-goel.github.io/4dhumans/.

Key facts. Cited by 344 papers in the map (data through 7 Oct 2026). Builds on 79 of its 102 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

79 of 102 references.

References · 79

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2017Mask R-CNNcited by 7,259
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2016Layer Normalizationcited by 5,579
  10. 2017The Kinetics Human Action Video Datasetcited by 1,894
  11. 2018SlowFast Networks for Video Recognitioncited by 1,862
  12. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
38 more
  1. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  2. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  3. 2021Multiscale Vision Transformerscited by 896
  4. 2017End-to-end Recovery of Human Shape and Posecited by 883
  5. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  6. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  7. 2016Performance Measures and a Data Set for Multi-Target, Multi-Camera Trackingcited by 685
  8. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  9. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  10. 2020HOTA: A Higher Order Metric for Evaluating Multi-Object Trackingcited by 546
  11. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 515
  12. 2016Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervisioncited by 493
  13. 2022HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Videocited by 483
  14. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
  15. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  16. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 417
  17. 2019Tracking without bells and whistlescited by 382
  18. 2016RMPE: Regional Multi-person Pose Estimationcited by 340
  19. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 334
  20. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  21. 2018Learning 3D Human Dynamics from Videocited by 315
  22. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  23. 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
  24. 2021Mesh Graphormercited by 264
  25. 2021PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loopcited by 250
  26. 2022CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimationcited by 237
  27. 2018Learning to Estimate 3D Human Pose and Shape from a Single Color Imagecited by 226
  28. 2017Unite the People: Closing the Loop Between 3D and 2D Human Representationscited by 205
  29. 2021AGORA: Avatars in Geography Optimized for Regression Analysiscited by 203
  30. 2018Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimationcited by 185
  31. 2017PoseTrack: A Benchmark for Human Pose Estimation and Trackingcited by 173
  32. 2020Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Videocited by 162
  33. 2021Probabilistic Modeling for Human Mesh Recoverycited by 152
  34. 2022PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Imagescited by 152
  35. 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 151
  36. 2021Putting People in their Place: Monocular Regression of 3D People in Depthcited by 141
  37. 2023Decoupling Human and Camera Motion from Videos in the Wildcited by 140
  38. 2015Human Pose Estimation with Iterative Error Feedbackcited by 121

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 103; 2025: 120; 2026: 104.

344 papers.

Sort:

By area:

  1. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  2. 2023Reconstructing Hands in 3D with Transformerscited by 298
  3. 2024Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidancecited by 271
  4. 2023WHAM: Reconstructing World-grounded Humans with Accurate 3D Motioncited by 200
  5. 2024OmniRe: Omni Urban Scene Reconstructioncited by 159
  6. 2023HUGS: Human Gaussian Splatscited by 154
  7. 2024World-Grounded Human Motion Recovery via Gravity-View Coordinatescited by 127
  8. 2023GART: Gaussian Articulated Template Modelscited by 121
  9. 2024WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wildcited by 117
  10. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 109
  11. 2024TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videoscited by 105
  12. 2024Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shotcited by 98
38 more
  1. 2024TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representationcited by 93
  2. 2023Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Renderingcited by 84
  3. 2024DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Controlcited by 80
  4. 2023ChatPose: Chatting about 3D Human Posecited by 71
  5. 2025PromptHMR: Promptable Human Mesh Recoverycited by 70
  6. 2024TAPVid-3D: A Benchmark for Tracking Any Point in 3Dcited by 69
  7. 2024SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Drivingcited by 66
  8. 2024CameraHMR: Aligning People with Perspectivecited by 64
  9. 2024MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modelingcited by 63
  10. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 62
  11. 2024Human4DiT: 360-degree Human Video Generation with 4D Diffusion Transformercited by 61
  12. 2026SAM 3D Body: Robust Full-Body Human Mesh Recoverycited by 61
  13. 2024Learning the 3D Fauna of the Webcited by 58
  14. 2024Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimationcited by 57
  15. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 57
  16. 2024Estimating Body and Hand Motion in an Ego-sensed Worldcited by 56
  17. 20244M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalitiescited by 52
  18. 2024RoHM: Robust Human Motion Reconstruction via Diffusioncited by 46
  19. 2024Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mambacited by 45
  20. 2023Generative Proxemics: A Prior for 3D Social Interaction from Imagescited by 43
  21. 2025GENMO: A GENeralist Model for Human MOtioncited by 43
  22. 2025LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instructioncited by 40
  23. 2024Score-Guided Diffusion for 3D Human Recoverycited by 39
  24. 2025DreamActor-M1: Holistic, Expressive and Robust Human Image Animation with Hybrid Guidancecited by 36
  25. 2025Human3R: Everyone Everywhere All at Oncecited by 35
  26. 2023HAP: Structure-Aware Masked Image Modeling for Human-Centric Perceptioncited by 33
  27. 2024Harmony4D: A Video Dataset for In-The-Wild Close Human Interactionscited by 31
  28. 2024Reconstructing People, Places, and Camerascited by 31
  29. 2024Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Cameracited by 30
  30. 2024MotionChain: Conversational Motion Controllers via Multimodal Promptscited by 30
  31. 2025HUMOTO: A 4D Dataset of Mocap Human Object Interactionscited by 30
  32. 2024Monocular Occupancy Prediction for Scalable Indoor Scenescited by 29
  33. 2024Unimotion: Unifying 3D Human Motion Synthesis and Understandingcited by 29
  34. 2025Joint Optimization for 4D Human-Scene Reconstruction in the Wildcited by 29
  35. 2024Spatial Cognition from Egocentric Video: Out of Sight, Not Out of Mindcited by 28
  36. 2024PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videoscited by 27
  37. 2024Towards Open Domain Text-Driven Synthesis of Multi-Person Motionscited by 27
  38. 2024Learning from Massive Human Videos for Universal Humanoid Pose Controlcited by 27

50 most cited of 344 All 344 (JSON)