mlsanityData to 7 Oct 2026

TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactions

arXiv 2412.06334 ↗First posted 9 Dec 2024cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

Modeling 3D human-object interaction (HOI) is a problem of great interest for computer vision and a key enabler for virtual and mixed-reality applications. Existing methods work in a one-way direction: some recover plausible human interactions conditioned on a 3D object; others recover the object pose conditioned on a human pose. Instead, we provide the first unified model - TriDi which works in any direction. Concretely, we generate Human, Object, and Interaction modalities simultaneously with a new three-way diffusion process, allowing to model seven distributions with one network. We implement TriDi as a transformer attending to the various modalities' tokens, thereby discovering conditional relations between them. The user can control the interaction either as a text description of HOI or a contact map. We embed these two representations into a shared latent space, combining the practicality of text descriptions with the expressiveness of contact maps. Using a single network, TriDi unifies all the special cases of prior work and extends to new ones, modeling a family of seven distributions. Remarkably, despite using a single model, TriDi generated samples surpass one-way specialized baselines on GRAB and BEHAVE in terms of both qualitative and quantitative metrics, and demonstrating better diversity. We show the applicability of TriDi to scene population, generating objects for human-contact datasets, and generalization to unseen object geometry. The project page is available at: https://virtualhumans.mpi-inf.mpg.de/tridi.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 94 of its 115 references that are in the map. First posted on arXiv 9 Dec 2024.

Builds on

94 of 115 references.

References · 94

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Video Diffusion Modelscited by 2,280
  8. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  9. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  10. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
  11. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  12. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
38 more
  1. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  2. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  3. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  4. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  5. 2022CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimationcited by 237
  6. 2023BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motioncited by 229
  7. 2023Object Motion Guided Human Motion Synthesiscited by 206
  8. 2021AGORA: Avatars in Geography Optimized for Regression Analysiscited by 203
  9. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  10. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 185
  11. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  12. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  13. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  14. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 176
  15. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  16. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  17. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  18. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 145
  19. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 142
  20. 2021Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensorscited by 137
  21. 2021ContactOpt: Optimizing Contact to Improve Graspscited by 137
  22. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  23. 2024Scaling Up Dynamic Human-Scene Interaction Modelingcited by 137
  24. 2020Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenescited by 135
  25. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 134
  26. 2022DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulationcited by 129
  27. 2019Generating 3D People in Scenes without Peoplecited by 127
  28. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 121
  29. 2015Capturing Hands in Action using Discriminative Salient Points and Physics Simulationcited by 115
  30. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 109
  31. 2023Controllable Human-Object Interaction Synthesiscited by 107
  32. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  33. 2022InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interactioncited by 90
  34. 2021SAGA: Stochastic Whole-Body Grasping with Contactcited by 89
  35. 2023Hi4D: 4D Instance Segmentation of Close Human Interactioncited by 87
  36. 2022Compositional Human-Scene Interaction Synthesis with Semantic Controlcited by 82
  37. 2024Generating Human Interaction Motions in Scenes with Text Controlcited by 80
  38. 2023NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesiscited by 78

50 most cited of 94 All 94 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 14.

18 papers.

Sort:

By area:

  1. 2026InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactionscited by 10
  2. 20253D Human Interaction Generation: A Surveycited by 6
  3. 2025HOSIG: Full-Body Human-Object-Scene Interaction Generation with Hierarchical Scene Perceptioncited by 5
  4. 2025ECHO: Ego-Centric modeling of Human-Object interactionscited by 4
  5. 2025TwoSquared: 4D Generation from 2D Image Pairscited by 3
  6. 2026Open-Vocabulary Functional 3D Human-Scene Interaction Generationcited by 2
  7. 2026ViHOI: Human-Object Interaction Synthesis with Visual Priorscited by 2
  8. 2026CLUTCH: Contextualized Language model for Unlocking Text-Conditioned Hand motion modelling in the wildcited by 1
  9. 2026Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUscited by 1
  10. 2026Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interactioncited by 1
  11. 2026Surface Keypoint Representation for Multi-Object and Articulated Human-Object Interaction Generationcited by 1
  12. 2026Ordered Diffusion for 3D Human Registrationcited by 1
6 more
  1. 2026HOIMask: Towards Generative Masked Modeling for Human Object Interaction Generationcited by 1
  2. 2026Unified Number-Free Text-to-Motion Generation Via Flow Matchingcited by 0
  3. 2026LEXIS: LatEnt ProXimal Interaction Signatures for 3D HOI from an Imagecited by 0
  4. 2026Reconstructing Humans and Objects in Interaction using Large Reconstruction Modelscited by 0
  5. 2026GraspHOI: Full-Body 3D Human-Object Reconstruction with Finger-Level Grasps from a Single In-the-Wild Imagecited by 0
  6. 2026Harnessing Coupled Stream Completion For Human-Object Interaction Modelingcited by 0