mlsanityData to 7 Oct 2026

BEHAVE: Dataset and Method for Tracking Human Object Interactions

arXiv 2204.06950 ↗First posted 14 Apr 2022cs.CVcited by 246

Not on the map. Why?

Cited by 31 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Modelling interactions between humans and objects in natural environments is central to many applications including gaming, virtual and mixed reality, as well as human behavior analysis and human-robot collaboration. This challenging operation scenario requires generalization to vast number of objects, scenes, and human actions. Unfortunately, there exist no such dataset. Moreover, this data needs to be acquired in diverse natural environments, which rules out 4D scanners and marker based capture systems. We present BEHAVE dataset, the first full body human- object interaction dataset with multi-view RGBD frames and corresponding 3D SMPL and object fits along with the annotated contacts between them. We record around 15k frames at 5 locations with 8 subjects performing a wide range of interactions with 20 common objects. We use this data to learn a model that can jointly track humans and objects in natural environments with an easy-to-use portable multi-camera setup. Our key insight is to predict correspondences from the human and the object to a statistical body model to obtain human-object contacts during interactions. Our approach can record and track not just the humans and objects but also their interactions, modeled as surface contacts, in 3D. Our code and data can be found at: http://virtualhumans.mpi-inf.mpg.de/behave

Key facts. Cited by 246 papers in the map (data through 7 Oct 2026). Builds on 69 of its 88 references that are in the map. First posted on arXiv 14 Apr 2022.

Builds on

69 of 88 references.

References · 69

  1. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  2. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  3. 2017End-to-end Recovery of Human Shape and Posecited by 883
  4. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  5. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  6. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  7. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  8. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  9. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 530
  10. 2020PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitizationcited by 521
  11. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  12. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
38 more
  1. 2020Implicit Functions in Feature Space for 3D Shape Reconstruction and Completioncited by 343
  2. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  3. 2020Neural Unsigned Distance Fields for Implicit Function Learningcited by 265
  4. 2021Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensorscited by 246
  5. 2019Multi-Garment Net: Learning to Dress 3D People from Imagescited by 244
  6. 2018Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodiescited by 239
  7. 2018Learning to Estimate 3D Human Pose and Shape from a Single Color Imagecited by 226
  8. 2018Pix3D: Dataset and Methods for Single-Image 3D Shape Modelingcited by 218
  9. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  10. 2020Long-term Human Motion Prediction with Scene Contextcited by 191
  11. 2021Character Controllers Using Motion VAEscited by 182
  12. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  13. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  14. 2019Learning to Reconstruct People in Clothing from a Single RGB Cameracited by 168
  15. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
  16. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  17. 2017MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstructioncited by 163
  18. 2020DeepCap: Monocular Human Performance Capture Using Weak Supervisioncited by 162
  19. 2019Learnable Triangulation of Human Posecited by 160
  20. 2021H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motioncited by 157
  21. 2020Monocular Expressive Body Regression through Body-Driven Attentioncited by 155
  22. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  23. 2021Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensorscited by 137
  24. 2021ContactOpt: Optimizing Contact to Improve Graspscited by 137
  25. 2021Collaborative Regression of Expressive Bodies using Moderationcited by 137
  26. 2020Combining Implicit Function Learning and Parametric Models for 3D Human Reconstructioncited by 133
  27. 2020CPF: Learning a Contact Potential Field to Model the Hand-Object Interactioncited by 127
  28. 2020f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentationcited by 117
  29. 2018Detailed Human Avatars from Monocular Videocited by 109
  30. 2017MarrNet: 3D Shape Reconstruction via 2.5D Sketchescited by 108
  31. 2020LoopReg: Self-supervised Learning of Implicit Surface Correspondences, Pose and Shape for 3D Human Mesh Registrationcited by 108
  32. 2021imGHUM: Implicit Generative Models of 3D Human Shape and Articulated Posecited by 102
  33. 2021Learning Motion Priors for 4D Human Body Capture in 3D Scenescited by 98
  34. 2017Towards Accurate Markerless Human Shape and Pose Estimation over Timecited by 92
  35. 2021On Self-Contact and Human Posecited by 88
  36. 2019ContactGrasp: Functional Multi-finger Grasp Synthesis from Contactcited by 87
  37. 2020Monocular Real-Time Volumetric Performance Capturecited by 74
  38. 2019Putting Humans in a Scene: Learning Affordance in 3D Indoor Environmentscited by 72

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 18; 2023: 45; 2024: 57; 2025: 74; 2026: 52.

246 papers.

Sort:

By area:

  1. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  2. 2023BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motioncited by 229
  3. 2023Object Motion Guided Human Motion Synthesiscited by 206
  4. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  5. 2023SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimationcited by 154
  6. 2023BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objectscited by 144
  7. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  8. 2024Scaling Up Dynamic Human-Scene Interaction Modelingcited by 137
  9. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 134
  10. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  11. 2024HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videoscited by 122
  12. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 121
38 more
  1. 2023Controllable Human-Object Interaction Synthesiscited by 107
  2. 2023EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wildcited by 95
  3. 2022InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interactioncited by 90
  4. 2023HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Modelscited by 89
  5. 2023Hi4D: 4D Instance Segmentation of Close Human Interactioncited by 87
  6. 2022Pose-NDF: Modeling Human Pose Manifolds with Neural Distance Fieldscited by 85
  7. 2023CIRCLE: Capture In Rich Contextual Environmentscited by 79
  8. 2023NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesiscited by 78
  9. 2024Seamless Human Motion Composition with Blended Positional Encodingscited by 77
  10. 2022Full-Body Articulated Human-Object Interactioncited by 76
  11. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 73
  12. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 72
  13. 2022TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinementcited by 69
  14. 2022NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactionscited by 67
  15. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 67
  16. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  17. 2024Human-Object Interaction from Human-Level Instructionscited by 62
  18. 2023Visibility Aware Human-Object Interaction Tracking from Single RGB Cameracited by 58
  19. 2022MIME: Human-Aware 3D Scene Generationcited by 57
  20. 2024Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimationcited by 57
  21. 2023DECO: Dense Estimation of 3D Human-Scene Contact In The Wildcited by 54
  22. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 54
  23. 2023ContactGen: Generative Contact Modeling for Grasp Generationcited by 52
  24. 2024HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environmentcited by 52
  25. 2022Learning 3D Human Pose Estimation from Dozens of Datasets using a Geometry-Aware Autoencoder to Bridge Between Skeleton Formatscited by 50
  26. 2023LEMON: Learning 3D Human-Object Interaction Relation from 2D Imagescited by 49
  27. 2024THOR: Text to Human-Object Interaction Diffusion via Relation Interventioncited by 46
  28. 2024Introducing HOT3D: An Egocentric Dataset for 3D Hand and Object Trackingcited by 46
  29. 2024The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motioncited by 46
  30. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 46
  31. 2024Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Modelscited by 44
  32. 2023I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactionscited by 42
  33. 2023Detecting Human-Object Contact in Imagescited by 41
  34. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 40
  35. 2023Generating Continual Human Motion in Diverse 3D Scenescited by 39
  36. 2022Differentiable Dynamics for Articulated 3d Human Motion Reconstructioncited by 38
  37. 2023Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generationcited by 37
  38. 2023Object pop-up: Can we infer 3D objects and their poses from human interactions alone?cited by 36

50 most cited of 246 All 246 (JSON)