# InteractMove: Text-Controlled Human-Object Interaction Generation in 3D Scenes with Movable Objects

arXiv: https://arxiv.org/abs/2509.23612 · HTML: https://mlsanity.com/paper/2509.23612/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 28 Sep 2025.

## Builds on (most cited first, 22 of 22)

- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [Diffusion-based Generation, Optimization, and Planning in 3D Scenes](https://mlsanity.com/paper/2301.06015/) (2023)
- [Object Motion Guided Human Motion Synthesis](https://mlsanity.com/paper/2309.16237/) (2023)
- [Stochastic Scene-Aware Motion Prediction](https://mlsanity.com/paper/2108.08284/) (2021)
- [InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion](https://mlsanity.com/paper/2308.16905/) (2023)
- [HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes](https://mlsanity.com/paper/2210.09729/) (2022)
- [GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping](https://mlsanity.com/paper/2112.11454/) (2021)
- [CG-HOI: Contact-Guided 3D Human-Object Interaction Generation](https://mlsanity.com/paper/2311.16097/) (2023)
- [Generating 3D People in Scenes without People](https://mlsanity.com/paper/1912.02923/) (2019)
- [Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction](https://mlsanity.com/paper/2404.00562/) (2024)
- [CIRCLE: Capture In Rich Contextual Environments](https://mlsanity.com/paper/2303.17912/) (2023)
- [Full-Body Articulated Human-Object Interaction](https://mlsanity.com/paper/2212.10621/) (2022)
- [Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments](https://mlsanity.com/paper/1903.05690/) (2019)
- [Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding](https://mlsanity.com/paper/2311.15383/) (2023)
- [Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memory](https://mlsanity.com/paper/2309.03696/) (2023)
- [Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection](https://mlsanity.com/paper/2404.06194/) (2024)
- [3D Vision and Language Pretraining with Large-Scale Synthetic Data](https://mlsanity.com/paper/2407.06084/) (2024)
- [PLACE: Proximity Learning of Articulation and Contact in 3D Environments](https://mlsanity.com/paper/2008.05570/) (2020)

## Built on by (most cited first, 3 of 3)

- [Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interaction](https://mlsanity.com/paper/2604.27491/) (2026)
- [Music-to-Dance Generation via Atomic Movements](https://mlsanity.com/paper/2607.13978/) (2026)
- [Puppeteer: Object-Grounded Posture-Aware Co-Speech Gesture Generation](https://mlsanity.com/paper/2609.00369/) (2026)

## Abstract (arXiv)

We propose a novel task of text-controlled human object interaction generation in 3D scenes with movable objects. Existing human-scene interaction datasets suffer from insufficient interaction categories and typically only consider interactions with static objects (do not change object positions), and the collection of such datasets with movable objects is difficult and costly. To address this problem, we construct the InteractMove dataset for Movable Human-Object Interaction in 3D Scenes by aligning existing human object interaction data with scene contexts, featuring three key characteristics: 1) scenes containing multiple movable objects with text-controlled interaction specifications (including same-category distractors requiring spatial and 3D scene context understanding), 2) diverse object types and sizes with varied interaction patterns (one-hand, two-hand, etc.), and 3) physically plausible object manipulation trajectories. With the introduction of various movable objects, this task becomes more challenging, as the model needs to identify objects to be interacted with accurately, learn to interact with objects of different sizes and categories, and avoid collisions between movable objects and the scene. To tackle such challenges, we propose a novel pipeline solution. We first use 3D visual grounding models to identify the interaction object. Then, we propose a hand-object joint affordance learning to predict contact regions for different hand joints and object parts, enabling accurate grasping and manipulation of diverse objects. Finally, we optimize interactions with local-scene modeling and collision avoidance constraints, ensuring physically plausible motions and avoiding collisions between objects and the scene. Comprehensive experiments demonstrate our method's superiority in generating physically plausible, text-compliant interactions compared to existing approaches.