# MAD-HOI: Masked Autoregressive Diffusion for Generating Articulated Hand Object Interactions from Text

arXiv: https://arxiv.org/abs/2608.10162 · HTML: https://mlsanity.com/paper/2608.10162/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 31 of its 41 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts](https://mlsanity.com/paper/2207.01696/) (2022)
- [Human Motion Diffusion as a Generative Prior](https://mlsanity.com/paper/2303.01418/) (2023)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions](https://mlsanity.com/paper/2304.05684/) (2023)
- [DLow: Diversifying Latent Flows for Diverse Human Motion Prediction](https://mlsanity.com/paper/2003.08386/) (2020)
- [AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System](https://mlsanity.com/paper/2307.04577/) (2023)
- [EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos](https://mlsanity.com/paper/2507.12440/) (2025)
- [TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding](https://mlsanity.com/paper/2401.08399/) (2024)
- [Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregression](https://mlsanity.com/paper/2411.16575/) (2024)
- [Humanoid Policy ~ Human Policy](https://mlsanity.com/paper/2503.13441/) (2025)
- [DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptions](https://mlsanity.com/paper/2403.17827/) (2024)
- [DexMachina: Functional Retargeting for Bimanual Dexterous Manipulation](https://mlsanity.com/paper/2505.24853/) (2025)
- [HOIGPT: Learning Long Sequence Hand-Object Interaction with Language Models](https://mlsanity.com/paper/2503.19157/) (2025)
- [BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objects](https://mlsanity.com/paper/2412.05066/) (2024)
- [DisCoRD: Discrete Tokens to Continuous Motion via Rectified Flow Decoding](https://mlsanity.com/paper/2411.19527/) (2024)
- [Auto-Regressive Diffusion for Generating 3D Human-Object Interactions](https://mlsanity.com/paper/2503.16801/) (2025)
- [OOD-HOI: Text-Driven 3D Whole-Body Human-Object Interactions Generation Beyond Training Domains](https://mlsanity.com/paper/2411.18660/) (2024)
- [GenHOI: Generalizing Text-driven 4D Human-Object Interaction Synthesis for Unseen Objects](https://mlsanity.com/paper/2506.15483/) (2025)

## Abstract (arXiv)

Methods for text-based generation of hand-object interaction (HOI) sequences primarily focus on producing smooth, physically plausible trajectories. A truly utilitarian method should additionally support variable-length generation, composite motion sequences, motion completion and infilling, and reliable termination without compromising physical plausibility. Standard diffusion models for HOI generation are typically trained only for text-to-motion generation on atomic motions and require the motion length to be specified a priori. Autoregressive (AR) methods provide greater sequence-level flexibility, but commonly depend on discrete motion codes, which can lose contact-sensitive motion detail. To address these key limitations, we present a model performing Masked Autoregression with Diffusion for HOI generation (MAD-HOI). Our method starts by encoding hand and object motions in a continuous latent space while keeping them disentangled to maintain stream-wise control. This is followed by a masked autoregressive transformer to predict context features that condition a flow-matching head. MAD-HOI is capable of motion generation for atomic and composite articulated sequences, conditioned motion completion and infilling, as well as EOM (End of Motion) prediction from a single training objective. We provide comprehensive evaluations for these capabilities and benchmark our method on the ARCTIC and GRAB datasets. Our experiments show that MAD-HOI achieves the strongest text-motion alignment and distributional generation quality against baselines on both ARCTIC and GRAB, while maintaining competitive geometric accuracy and physical plausibility.