mlsanityData to 7 Oct 2026

InterTrack: Tracking Human Object Interaction without Object Templates

arXiv 2408.13953 ↗First posted 25 Aug 2024cs.CVcited by 22

Not on the map. Why?

Abstract (arXiv)

Tracking human object interaction from videos is important to understand human behavior from the rapidly growing stream of video data. Previous video-based methods require predefined object templates while single-image-based methods are template-free but lack temporal consistency. In this paper, we present a method to track human object interaction without any object shape templates. We decompose the 4D tracking problem into per-frame pose tracking and canonical shape optimization. We first apply a single-view reconstruction method to obtain temporally-inconsistent per-frame interaction reconstructions. Then, for the human, we propose an efficient autoencoder to predict SMPL vertices directly from the per-frame reconstructions, introducing temporally consistent correspondence. For the object, we introduce a pose estimator that leverages temporal information to predict smooth object rotations under occlusions. To train our model, we propose a method to generate synthetic interaction videos and synthesize in total 10 hour videos of 8.5k sequences with full 3D ground truth. Experiments on BEHAVE and InterCap show that our method significantly outperforms previous template-based video tracking and single-frame reconstruction methods. Our proposed synthetic video dataset also allows training video-based methods that generalize to real-world videos. Our code and dataset will be publicly released.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 92 of its 107 references that are in the map. First posted on arXiv 25 Aug 2024.

Builds on

92 of 107 references.

References · 92

  1. 2017Attention Is All You Needcited by 59,932
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  3. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  4. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  5. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  6. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  7. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  8. 2022HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Videocited by 483
  9. 2023FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objectscited by 411
  10. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  11. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
  12. 2019Point-Voxel CNN for Efficient 3D Deep Learningcited by 343
38 more
  1. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 334
  2. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  3. 2018DeepIM: Deep Iterative Matching for 6D Pose Estimationcited by 283
  4. 2019What Do Single-view 3D Reconstruction Networks Learn?cited by 259
  5. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  6. 2019Multi-Garment Net: Learning to Dress 3D People from Imagescited by 244
  7. 2023Object Motion Guided Human Motion Synthesiscited by 206
  8. 2021BANMo: Building Animatable 3D Neural Models from Many Casual Videoscited by 195
  9. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 176
  10. 20183D-CODED : 3D Correspondences by Deep Deformationcited by 160
  11. 2022OnePose: One-Shot Object Pose Estimation without CAD Modelscited by 155
  12. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  13. 2023BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objectscited by 144
  14. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  15. 2021Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensorscited by 137
  16. 2017Deep Functional Maps: Structured Prediction for Dense Shape Correspondencecited by 134
  17. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 134
  18. 2020Combining Implicit Function Learning and Parametric Models for 3D Human Reconstructioncited by 133
  19. 2021LASR: Learning Articulated Shape Reconstruction from a Monocular Videocited by 115
  20. 2023GigaPose: Fast and Robust Novel Object Pose Estimation via One Correspondencecited by 113
  21. 2022MagicPony: Learning Articulated 3D Animals in the Wildcited by 111
  22. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  23. 2020LoopReg: Self-supervised Learning of Implicit Surface Correspondences, Pose and Shape for 3D Human Mesh Registrationcited by 108
  24. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  25. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  26. 2022InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interactioncited by 90
  27. 2021SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimationcited by 88
  28. 2021BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Modelscited by 87
  29. 2020Deep Implicit Templates for 3D Shape Representationcited by 86
  30. 2022Pose-NDF: Modeling Human Pose Manifolds with Neural Distance Fieldscited by 85
  31. 2023PC2: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstructioncited by 84
  32. 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 76
  33. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 73
  34. 2022TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinementcited by 69
  35. 2022NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactionscited by 67
  36. 2023DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objectscited by 64
  37. 2023Neural Parametric Gaussians for Monocular Non-Rigid Object Reconstructioncited by 60
  38. 2023Visibility Aware Human-Object Interaction Tracking from Single RGB Cameracited by 58

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 10.

22 papers.

Sort:

By area:

  1. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 67
  2. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 46
  3. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 31
  4. 2025HERO: Human Reaction Generation from Videoscited by 15
  5. 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 14
  6. 2025CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interactioncited by 14
  7. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 13
  8. 2025PhySIC: Physically Plausible 3D Human-Scene Interaction and Contact from a Single Imagecited by 13
  9. 2026InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactionscited by 10
  10. 2025MVGBench: Comprehensive Benchmark for Multi-view Generation Modelscited by 9
  11. 2026Unleashing Guidance Without Classifiers for Human-Object Interaction Animationcited by 5
  12. 2025ECHO: Ego-Centric modeling of Human-Object interactionscited by 4
10 more
  1. 2026End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstructioncited by 1
  2. 2026InterPhys: Physics-aware Human Motion Synthesis in a Dynamic Scenecited by 1
  3. 2026Scene and Human in One World: Reconstruction in a Feedforward Passcited by 1
  4. 2025Realistic Clothed Human and Object Joint Reconstruction from a Single Imagecited by 0
  5. 2025Human Motion Estimation with Everyday Wearablescited by 0
  6. 2026SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergycited by 0
  7. 2026Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videoscited by 0
  8. 2026RHINO: Reconstructing Human Interactions with Novel Objects from Monocular Videoscited by 0
  9. 2026Recovering Physically Plausible Human-Object Interactions from Monocular Videoscited by 0
  10. 2026GraspHOI: Full-Body 3D Human-Object Reconstruction with Finger-Level Grasps from a Single In-the-Wild Imagecited by 0