mlsanityData to 14 Oct 2026

The One Where They Reconstructed 3D Humans and Environments in TV Shows

arXiv 2207.14279 ↗First posted 28 Jul 2022cs.CVcited by 30

Not on the map. Why?

Abstract (arXiv)

TV shows depict a wide variety of human behaviors and have been studied extensively for their potential to be a rich source of data for many applications. However, the majority of the existing work focuses on 2D recognition tasks. In this paper, we make the observation that there is a certain persistence in TV shows, i.e., repetition of the environments and the humans, which makes possible the 3D reconstruction of this content. Building on this insight, we propose an automatic approach that operates on an entire season of a TV show and aggregates information in 3D; we build a 3D model of the environment, compute camera information, static 3D scene structure and body scale information. Then, we demonstrate how this information acts as rich 3D context that can guide and improve the recovery of 3D human pose and position in these environments. Moreover, we show that reasoning about humans and their environment in 3D enables a broad range of downstream applications: re-identification, gaze estimation, cinematography and image editing. We apply our approach on environments from seven iconic TV shows and perform an extensive evaluation of the proposed system.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 50 of its 79 references that are in the map. First posted on arXiv 28 Jul 2022.

Builds on

50 of 79 references.

References · 50

  1. 2017Mask R-CNNcited by 7,277
  2. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  3. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,232
  4. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,198
  5. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,162
  6. 2021Volume Rendering of Neural Implicit Surfacescited by 996
  7. 2017End-to-end Recovery of Human Shape and Posecited by 885
  8. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 751
  9. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 695
  10. 2021UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstructioncited by 646
  11. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 627
  12. 2021Dynamic View Synthesis from Dynamic Monocular Videocited by 420
38 more
  1. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 337
  2. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 324
  3. 2020DISK: Learning local features with policy gradientcited by 312
  4. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 295
  5. 2018DeepMVS: Learning Multi-view Stereopsiscited by 277
  6. 2020Consistent Video Depth Estimationcited by 220
  7. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  8. 2019Learning Individual Styles of Conversational Gesturecited by 191
  9. 2020Robust Consistent Video Depth Estimationcited by 188
  10. 2019Learnable Triangulation of Human Posecited by 161
  11. 2021Probabilistic Modeling for Human Mesh Recoverycited by 153
  12. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  13. 2021Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensorscited by 137
  14. 2020Coherent Reconstruction of Multiple Humans from a Single Imagecited by 136
  15. 2021SPEC: Seeing People in the Wild with an Estimated Cameracited by 135
  16. 2021SimPoE: Simulated Character Control for 3D Human Pose Estimationcited by 114
  17. 2021Learning Motion Priors for 4D Human Body Capture in 3D Scenescited by 98
  18. 2020Contact and Human Dynamics from Monocular Videocited by 95
  19. 2020Unsupervised Pre-training for Person Re-identificationcited by 95
  20. 2017Towards Accurate Markerless Human Shape and Pose Estimation over Timecited by 92
  21. 2019Fast and Robust Multi-Person 3D Pose Estimation from Multiple Viewscited by 83
  22. 2021Neural Monocular 3D Human Motion Capture with Physical Awarenesscited by 71
  23. 2021Physics-based Human Motion Estimation and Synthesis from Videoscited by 66
  24. 2020Holistic 3D Human and Scene Mesh Estimation from Single View Imagescited by 54
  25. 2020Human Mesh Recovery from Multiple Shotscited by 52
  26. 20204D Human Body Capture from Egocentric Video via 3D Scene Groundingcited by 43
  27. 2018Binge Watching: Scaling Affordance Learning from Sitcomscited by 41
  28. 2021D3D-HOI: Dynamic 3D Human-Object Interactions from Videoscited by 39
  29. 2017MonoCap: Monocular Human Motion Capture using a CNN Coupled with a Geometric Priorcited by 36
  30. 2019LAEO-Net: revisiting people Looking At Each Other in videoscited by 34
  31. 2020Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamicscited by 34
  32. 2020Reconstructing NBA Playerscited by 24
  33. 2021Face, Body, Voice: Video Person-Clustering with Multiple Modalitiescited by 24
  34. 2018From Benedict Cumberbatch to Sherlock Holmes: Character Identification in TV series without a Scriptcited by 20
  35. 2018Soccer on Your Tabletopcited by 20
  36. 2019Video Face Clustering with Unknown Number of Clusterscited by 17
  37. 2018Person Search in Videos with One Portrait Through Visual and Temporal Linkscited by 14
  38. 2019Temporally Coherent General Dynamic Scene Reconstructioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 13; 2024: 8; 2025: 4; 2026: 4.

30 papers.

Sort:

By area:

  1. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 600
  2. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 350
  3. 2023Decoupling Human and Camera Motion from Videos in the Wildcited by 142
  4. 2024TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videoscited by 106
  5. 2023TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environmentscited by 91
  6. 2024Estimating Body and Hand Motion in an Ego-sensed Worldcited by 58
  7. 2023AutoAD: Movie Description in Contextcited by 50
  8. 2023Learning Disentangled Avatars with Hybrid 3D Representationscited by 44
  9. 2025Human3R: Everyone Everywhere All at Oncecited by 37
  10. 2023PACE: Human and Camera Motion Estimation from in-the-wild Videoscited by 34
  11. 2022Scene-aware Egocentric 3D Human Pose Estimationcited by 30
  12. 2025Joint Optimization for 4D Human-Scene Reconstruction in the Wildcited by 29
18 more
  1. 2024Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questionscited by 22
  2. 2023RICO: Regularizing the Unobservable for Indoor Compositional Reconstructioncited by 20
  3. 2023Total-Recon: Deformable Scene Reconstruction for Embodied View Synthesiscited by 20
  4. 2024Cinematographic Camera Diffusion Modelcited by 18
  5. 2024COIN: Control-Inpainting Diffusion Prior for Human and Camera Motion Estimationcited by 15
  6. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 14
  7. 2023Cinematic Behavior Transfer via NeRF-based Differentiable Filmingcited by 12
  8. 2024AWOL: Analysis WithOut synthesis using Languagecited by 12
  9. 2026UniSH: Unifying Scene and Human Reconstruction in a Feed-Forward Passcited by 9
  10. 2023Long-range Multimodal Pretraining for Movie Understandingcited by 7
  11. 2024Toon3D: Seeing Cartoons from New Perspectivescited by 3
  12. 2024Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videoscited by 3
  13. 2023Mesh Strikes Back: Fast and Efficient Human Reconstruction from RGB videoscited by 2
  14. 2026Hand3R: Online 4D Hand-Scene Reconstruction in the Wildcited by 1
  15. 2026UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Videocited by 1
  16. 2023BluNF: Blueprint Neural Fieldcited by 0
  17. 2025ShowMak3r: Compositional TV Show Reconstructioncited by 0
  18. 2026Multi-THuMBS: Multi-person Tracking of 3D Human Meshes Beyond Video Shotscited by 0