mlsanityData to 14 Oct 2026

InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interaction

arXiv 2209.12354 ↗First posted 26 Sep 2022cs.CVcited by 93

Not on the map. Why?

Abstract (arXiv)

Humans constantly interact with daily objects to accomplish tasks. To understand such interactions, computers need to reconstruct these from cameras observing whole-body interaction with scenes. This is challenging due to occlusion between the body and objects, motion blur, depth/scale ambiguities, and the low image resolution of hands and graspable object parts. To make the problem tractable, the community focuses either on interacting hands, ignoring the body, or on interacting bodies, ignoring hands. The GRAB dataset addresses dexterous whole-body interaction but uses marker-based MoCap and lacks images, while BEHAVE captures video of body object interaction but lacks hand detail. We address the limitations of prior work with InterCap, a novel method that reconstructs interacting whole-bodies and objects from multi-view RGB-D data, using the parametric whole-body model SMPL-X and known object meshes. To tackle the above challenges, InterCap uses two key observations: (i) Contact between the hand and object can be used to improve the pose estimation of both. (ii) Azure Kinect sensors allow us to set up a simple multi-view RGB-D capture system that minimizes the effect of occlusion while providing reasonable inter-camera synchronization. With this method we capture the InterCap dataset, which contains 10 subjects (5 males and 5 females) interacting with 10 objects of various sizes and affordances, including contact with the hands or feet. In total, InterCap has 223 RGB-D videos, resulting in 67,357 multi-view frames, each containing 6 RGB-D images. Our method provides pseudo ground-truth body meshes and objects for each video frame. Our InterCap method and dataset fill an important gap in the literature and support many research directions. Our data and code are areavailable for research purposes.

Key facts. Cited by 93 papers in the map (data through 14 Oct 2026). Builds on 55 of its 98 references that are in the map. First posted on arXiv 26 Sep 2022.

Builds on

55 of 98 references.

References · 55

  1. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  2. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
  3. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,162
  4. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 915
  5. 2017End-to-end Recovery of Human Shape and Posecited by 885
  6. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 695
  7. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 627
  8. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 620
  9. 2019PointRend: Image Segmentation as Renderingcited by 440
  10. 2017Neural 3D Mesh Renderercited by 437
  11. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
  12. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
38 more
  1. 2016Convolutional Pose Machinescited by 325
  2. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 324
  3. 2018Learning 3D Human Dynamics from Videocited by 316
  4. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 312
  5. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 295
  6. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 249
  7. 2018Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodiescited by 239
  8. 2018Video Based Reconstruction of 3D People Modelscited by 238
  9. 2020STAR: Sparse Trained Articulated Human Body Regressorcited by 194
  10. 2020Long-term Human Motion Prediction with Scene Contextcited by 192
  11. 2018Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimationcited by 185
  12. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 183
  13. 2022Embodied Hands: Modeling and Capturing Hands and Bodies Togethercited by 181
  14. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 180
  15. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 179
  16. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  17. 2019Learnable Triangulation of Human Posecited by 161
  18. 2022OnePose: One-Shot Object Pose Estimation without CAD Modelscited by 157
  19. 2020Monocular Expressive Body Regression through Body-Driven Attentioncited by 155
  20. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  21. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 144
  22. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  23. 2017VNect: Real-time 3D Human Pose Estimation with a Single RGB Cameracited by 129
  24. 2019Generating 3D People in Scenes without Peoplecited by 127
  25. 2020Epipolar Transformerscited by 117
  26. 2015Capturing Hands in Action using Discriminative Salient Points and Physics Simulationcited by 115
  27. 2022Recovering 3D Human Mesh from Monocular Images: A Surveycited by 103
  28. 2021Learning Motion Priors for 4D Human Body Capture in 3D Scenescited by 98
  29. 2017Towards Accurate Markerless Human Shape and Pose Estimation over Timecited by 92
  30. 2021SAGA: Stochastic Whole-Body Grasping with Contactcited by 91
  31. 2019Cross View Fusion for 3D Human Pose Estimationcited by 90
  32. 2016Long-term Temporal Convolutions for Action Recognitioncited by 84
  33. 2019Fast and Robust Multi-Person 3D Pose Estimation from Multiple Viewscited by 83
  34. 2019Putting Humans in a Scene: Learning Affordance in 3D Indoor Environmentscited by 72
  35. 2017MonoPerfCap: Human Performance Capture from Monocular Videocited by 63
  36. 2022Human-Aware Object Placement for Visual Environment Reconstructioncited by 56
  37. 20204D Association Graph for Realtime Multi-person Motion Capture Using Multiple Video Camerascited by 49
  38. 2020Detecting Hands and Recognizing Physical Contact in the Wildcited by 38

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 16; 2024: 19; 2025: 32; 2026: 24.

93 papers.

Sort:

By area:

  1. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 182
  2. 2023BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objectscited by 148
  3. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 124
  4. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 76
  5. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 72
  6. 2022NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactionscited by 68
  7. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 64
  8. 2024Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimationcited by 59
  9. 2023Visibility Aware Human-Object Interaction Tracking from Single RGB Cameracited by 58
  10. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 57
  11. 2023DECO: Dense Estimation of 3D Human-Scene Contact In The Wildcited by 54
  12. 2024HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environmentcited by 53
38 more
  1. 2023LEMON: Learning 3D Human-Object Interaction Relation from 2D Imagescited by 49
  2. 2024THOR: Text to Human-Object Interaction Diffusion via Relation Interventioncited by 46
  3. 2023I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactionscited by 44
  4. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 44
  5. 2023Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generationcited by 38
  6. 2023Object pop-up: Can we infer 3D objects and their poses from human interactions alone?cited by 37
  7. 2024HIMO: A New Benchmark for Full-Body Human Interacting with Multiple Objectscited by 36
  8. 2024Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Modelscited by 32
  9. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 32
  10. 2025HUMOTO: A 4D Dataset of Mocap Human Object Interactionscited by 31
  11. 2024Joint Reconstruction of 3D Human and Object via Contact-Based Refinement Transformercited by 29
  12. 2023GRIP: Generating Interaction Poses Using Spatial Cues and Latent Consistencycited by 27
  13. 2025HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generationcited by 26
  14. 2025A Survey on Human Interaction Motion Generationcited by 25
  15. 2024InterTrack: Tracking Human Object Interaction without Object Templatescited by 23
  16. 2025PICO: Reconstructing 3D People In Contact with Objectscited by 22
  17. 2023NSF: Neural Surface Fields for Human Modeling from Monocular Depthcited by 21
  18. 2025InteractVLM: 3D Interaction Reasoning from 2D Foundational Modelscited by 21
  19. 2024GEARS: Local Geometry-aware Hand-object Interaction Synthesiscited by 20
  20. 2024TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactionscited by 20
  21. 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 16
  22. 2024Dense Hand-Object(HO) GraspNet with Full Grasping Taxonomy and Dynamicscited by 15
  23. 2024MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotationscited by 14
  24. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 14
  25. 2025CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interactioncited by 14
  26. 2025Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactionscited by 13
  27. 2026InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactionscited by 11
  28. 2024RHOBIN Challenge: Reconstruction of Human Object Interactioncited by 10
  29. 2023Compositional 3D Human-Object Neural Animationcited by 9
  30. 2025MVHumanNet++: A Large-scale Dataset of Multi-view Daily Dressing Human Captures with Richer Annotations for 3D Human Digitizationcited by 9
  31. 2025ScoreHOI: Physically Plausible Reconstruction of Human-Object Interaction via Score-Guided Diffusioncited by 9
  32. 2025End-to-End HOI Reconstruction Transformer with Graph-based Encodingcited by 8
  33. 2025Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactionscited by 8
  34. 2023HOH: Markerless Multimodal Human-Object-Human Handover Dataset with Large Object Countcited by 7
  35. 2023Ins-HOI: Instance Aware Human-Object Interactions Recoverycited by 7
  36. 2023Primitive-based 3D Human-Object Interaction Modelling and Programmingcited by 7
  37. 2024FORCE: Physics-aware Human-object Interactioncited by 7
  38. 20253D Human Interaction Generation: A Surveycited by 7

50 most cited of 93 All 93 (JSON)