# TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactions

arXiv: https://arxiv.org/abs/2412.06334 · HTML: https://mlsanity.com/paper/2412.06334/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 94 of its 115 references that are in the map.
- First posted on arXiv 9 Dec 2024.

## Builds on (most cited first, 30 of 94)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies](https://mlsanity.com/paper/2206.04670/) (2022)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation](https://mlsanity.com/paper/2208.00571/) (2022)
- [BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion](https://mlsanity.com/paper/2306.16940/) (2023)
- [Object Motion Guided Human Motion Synthesis](https://mlsanity.com/paper/2309.16237/) (2023)
- [AGORA: Avatars in Geography Optimized for Regression Analysis](https://mlsanity.com/paper/2104.14643/) (2021)
- [Stochastic Scene-Aware Motion Prediction](https://mlsanity.com/paper/2108.08284/) (2021)
- [One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale](https://mlsanity.com/paper/2303.06555/) (2023)
- [ContactPose: A Dataset of Grasps with Object Contact and Hand Pose](https://mlsanity.com/paper/2007.09545/) (2020)
- [InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion](https://mlsanity.com/paper/2308.16905/) (2023)
- [Capturing and Inferring Dense Full-Body Human-Scene Contact](https://mlsanity.com/paper/2206.09553/) (2022)
- [Grasping Field: Learning Implicit Representations for Human Grasps](https://mlsanity.com/paper/2008.04451/) (2020)
- [Populating 3D Scenes by Learning Human-Scene Interaction](https://mlsanity.com/paper/2012.11581/) (2020)
- [Hand-Object Contact Consistency Reasoning for Human Grasps Generation](https://mlsanity.com/paper/2104.03304/) (2021)
- [Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild](https://mlsanity.com/paper/2007.15649/) (2020)
- [ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging](https://mlsanity.com/paper/1904.06830/) (2019)

## Built on by (most cited first, 20 of 20)

- [InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactions](https://mlsanity.com/paper/2602.06035/) (2026)
- [3D Human Interaction Generation: A Survey](https://mlsanity.com/paper/2503.13120/) (2025)
- [HOSIG: Full-Body Human-Object-Scene Interaction Generation with Hierarchical Scene Perception](https://mlsanity.com/paper/2506.01579/) (2025)
- [ECHO: Ego-Centric modeling of Human-Object interactions](https://mlsanity.com/paper/2508.21556/) (2025)
- [TwoSquared: 4D Generation from 2D Image Pairs](https://mlsanity.com/paper/2504.12825/) (2025)
- [Open-Vocabulary Functional 3D Human-Scene Interaction Generation](https://mlsanity.com/paper/2601.20835/) (2026)
- [ViHOI: Human-Object Interaction Synthesis with Visual Priors](https://mlsanity.com/paper/2603.24383/) (2026)
- [Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUs](https://mlsanity.com/paper/2604.21926/) (2026)
- [Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interaction](https://mlsanity.com/paper/2604.27491/) (2026)
- [Surface Keypoint Representation for Multi-Object and Articulated Human-Object Interaction Generation](https://mlsanity.com/paper/2608.03158/) (2026)
- [HOIMask: Towards Generative Masked Modeling for Human Object Interaction Generation](https://mlsanity.com/paper/2608.15141/) (2026)
- [CLUTCH: Contextualized Language model for Unlocking Text-Conditioned Hand motion modelling in the wild](https://mlsanity.com/paper/2602.17770/) (2026)
- [Ordered Diffusion for 3D Human Registration](https://mlsanity.com/paper/2608.05804/) (2026)
- [Unified Number-Free Text-to-Motion Generation Via Flow Matching](https://mlsanity.com/paper/2603.27040/) (2026)
- [LEXIS: LatEnt ProXimal Interaction Signatures for 3D HOI from an Image](https://mlsanity.com/paper/2604.20800/) (2026)
- [Reconstructing Humans and Objects in Interaction using Large Reconstruction Models](https://mlsanity.com/paper/2608.27407/) (2026)
- [GraspHOI: Full-Body 3D Human-Object Reconstruction with Finger-Level Grasps from a Single In-the-Wild Image](https://mlsanity.com/paper/2608.28386/) (2026)
- [Harnessing Coupled Stream Completion For Human-Object Interaction Modeling](https://mlsanity.com/paper/2609.32551/) (2026)
- [A Benchmark for Spatially Grounded Gesture Generation](https://mlsanity.com/paper/2610.03105/) (2026)
- [COSMI: COmpositional Synthesis of Multi-object Interactions](https://mlsanity.com/paper/2610.03252/) (2026)

## Abstract (arXiv)

Modeling 3D human-object interaction (HOI) is a problem of great interest for computer vision and a key enabler for virtual and mixed-reality applications. Existing methods work in a one-way direction: some recover plausible human interactions conditioned on a 3D object; others recover the object pose conditioned on a human pose. Instead, we provide the first unified model - TriDi which works in any direction. Concretely, we generate Human, Object, and Interaction modalities simultaneously with a new three-way diffusion process, allowing to model seven distributions with one network. We implement TriDi as a transformer attending to the various modalities' tokens, thereby discovering conditional relations between them. The user can control the interaction either as a text description of HOI or a contact map. We embed these two representations into a shared latent space, combining the practicality of text descriptions with the expressiveness of contact maps. Using a single network, TriDi unifies all the special cases of prior work and extends to new ones, modeling a family of seven distributions. Remarkably, despite using a single model, TriDi generated samples surpass one-way specialized baselines on GRAB and BEHAVE in terms of both qualitative and quantitative metrics, and demonstrating better diversity. We show the applicability of TriDi to scene population, generating objects for human-contact datasets, and generalization to unseen object geometry. The project page is available at: https://virtualhumans.mpi-inf.mpg.de/tridi.