# Auto-Regressive Diffusion for Generating 3D Human-Object Interactions

arXiv: https://arxiv.org/abs/2503.16801 · HTML: https://mlsanity.com/paper/2503.16801/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 44 references that are in the map.
- First posted on arXiv 21 Mar 2025.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts](https://mlsanity.com/paper/2207.01696/) (2022)
- [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019)
- [MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis](https://mlsanity.com/paper/2212.04495/) (2022)
- [FLAME: Free-form Language-based Motion Synthesis & Editing](https://mlsanity.com/paper/2209.00349/) (2022)
- [Object Motion Guided Human Motion Synthesis](https://mlsanity.com/paper/2309.16237/) (2023)
- [InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions](https://mlsanity.com/paper/2304.05684/) (2023)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion](https://mlsanity.com/paper/2308.16905/) (2023)
- [Playing for Benchmarks](https://mlsanity.com/paper/1709.07322/) (2017)
- [Populating 3D Scenes by Learning Human-Scene Interaction](https://mlsanity.com/paper/2012.11581/) (2020)
- [HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes](https://mlsanity.com/paper/2210.09729/) (2022)
- [CG-HOI: Contact-Guided 3D Human-Object Interaction Generation](https://mlsanity.com/paper/2311.16097/) (2023)
- [AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanism](https://mlsanity.com/paper/2309.00796/) (2023)
- [Motion Mamba: Efficient and Long Sequence Motion Generation](https://mlsanity.com/paper/2403.07487/) (2024)

## Built on by (most cited first, 12 of 12)

- [InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactions](https://mlsanity.com/paper/2602.06035/) (2026)
- [HOI-Dyn: Learning Interaction Dynamics for Human-Object Motion Diffusion](https://mlsanity.com/paper/2507.01737/) (2025)
- [Pulp Motion: Framing-aware multimodal camera and human motion generation](https://mlsanity.com/paper/2510.05097/) (2025)
- [GenHOI: Generalizing Text-driven 4D Human-Object Interaction Synthesis for Unseen Objects](https://mlsanity.com/paper/2506.15483/) (2025)
- [ARMFlow: AutoRegressive MeanFlow for Online 3D Human Reaction Generation](https://mlsanity.com/paper/2512.16234/) (2025)
- [Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interaction](https://mlsanity.com/paper/2604.27491/) (2026)
- [UniHM: Universal Human Motion Generation with Object Interactions in Indoor Scenes](https://mlsanity.com/paper/2505.12774/) (2025)
- [Dynamic Full-body Motion Agent with Object Interaction via Blending Pre-trained Modular Controllers](https://mlsanity.com/paper/2605.11369/) (2026)
- [Policy-as-Data: Learning Generalizable HOI Diffusion Models from Simulated Physics](https://mlsanity.com/paper/2606.22806/) (2026)
- [MAD-HOI: Masked Autoregressive Diffusion for Generating Articulated Hand Object Interactions from Text](https://mlsanity.com/paper/2608.10162/) (2026)
- [Harnessing Coupled Stream Completion For Human-Object Interaction Modeling](https://mlsanity.com/paper/2609.32551/) (2026)
- [PAMI: Part Anchored Motion for Text to Human-Object Interaction Generation](https://mlsanity.com/paper/2609.38466/) (2026)

## Abstract (arXiv)

Text-driven Human-Object Interaction (Text-to-HOI) generation is an emerging field with applications in animation, video games, virtual reality, and robotics. A key challenge in HOI generation is maintaining interaction consistency in long sequences. Existing Text-to-Motion-based approaches, such as discrete motion tokenization, cannot be directly applied to HOI generation due to limited data in this domain and the complexity of the modality. To address the problem of interaction consistency in long sequences, we propose an autoregressive diffusion model (ARDHOI) that predicts the next continuous token. Specifically, we introduce a Contrastive Variational Autoencoder (cVAE) to learn a physically plausible space of continuous HOI tokens, thereby ensuring that generated human-object motions are realistic and natural. For generating sequences autoregressively, we develop a Mamba-based context encoder to capture and maintain consistent sequential actions. Additionally, we implement an MLP-based denoiser to generate the subsequent token conditioned on the encoded context. Our model has been evaluated on the OMOMO and BEHAVE datasets, where it outperforms existing state-of-the-art methods in terms of both performance and inference speed. This makes ARDHOI a robust and efficient solution for text-driven HOI tasks