# HumanVLA: Towards Vision-Language Directed Object Rearrangement by Physical Humanoid

arXiv: https://arxiv.org/abs/2406.19972 · HTML: https://mlsanity.com/paper/2406.19972/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2024.

## Built on by (most cited first, 19 of 19)

- [TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization](https://mlsanity.com/paper/2503.19901/) (2025)
- [Humanoid-VLA: Towards Universal Humanoid Control with Visual Integration](https://mlsanity.com/paper/2502.14795/) (2025)
- [Design2GarmentCode: Turning Design Concepts to Tangible Garments Through Program Synthesis](https://mlsanity.com/paper/2412.08603/) (2024)
- [Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactions](https://mlsanity.com/paper/2503.15898/) (2025)
- [Human-Centric Foundation Models: Perception, Generation and Agentic Modeling](https://mlsanity.com/paper/2502.08556/) (2025)
- [Human-Object Interaction via Automatically Designed VLM-Guided Motion Policy](https://mlsanity.com/paper/2503.18349/) (2025)
- [Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions](https://mlsanity.com/paper/2508.04681/) (2025)
- [Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learning](https://mlsanity.com/paper/2505.12278/) (2025)
- [AGILE: A Comprehensive Workflow for Humanoid Loco-Manipulation Learning](https://mlsanity.com/paper/2603.20147/) (2026)
- [InfBaGel: Human-Object-Scene Interaction Generation with Dynamic Perception and Iterative Refinement](https://mlsanity.com/paper/2604.04843/) (2026)
- [Endowing GPT-4 with a Humanoid Body: Building the Bridge Between Off-the-Shelf VLMs and the Physical World](https://mlsanity.com/paper/2511.00041/) (2025)
- [VCA: Vision-Click-Action Framework for Precise Manipulation of Segmented Objects in Target Ambiguous Environments](https://mlsanity.com/paper/2602.23583/) (2026)
- [GBC: Generalized Behavior-Cloning Framework for Whole-Body Humanoid Imitation](https://mlsanity.com/paper/2508.09960/) (2025)
- [LHM-Humanoid: Long-Horizon Human Motion Control for Continuous Object Transport in Cluttered Scenes](https://mlsanity.com/paper/2508.16943/) (2025)
- [Learn Weightlessness: Imitate Non-Self-Stabilizing Motions on Humanoid Robot](https://mlsanity.com/paper/2604.21351/) (2026)
- [Towards Highly-Constrained Human Motion Generation with Retrieval-Guided Diffusion Noise Optimization](https://mlsanity.com/paper/2605.08054/) (2026)
- [Dynamic Full-body Motion Agent with Object Interaction via Blending Pre-trained Modular Controllers](https://mlsanity.com/paper/2605.11369/) (2026)
- [Opt2VLA: Force-Aware Vision-Language-Action for Contact-Rich Humanoid Whole-Body Manipulation](https://mlsanity.com/paper/2609.23968/) (2026)
- [Are Vision-Language-Action Models Robust to One-Step Observation Perturbations?](https://mlsanity.com/paper/2609.32550/) (2026)

## Abstract (arXiv)

Physical Human-Scene Interaction (HSI) plays a crucial role in numerous applications. However, existing HSI techniques are limited to specific object dynamics and privileged information, which prevents the development of more comprehensive applications. To address this limitation, we introduce HumanVLA for general object rearrangement directed by practical vision and language. A teacher-student framework is utilized to develop HumanVLA. A state-based teacher policy is trained first using goal-conditioned reinforcement learning and adversarial motion prior. Then, it is distilled into a vision-language-action model via behavior cloning. We propose several key insights to facilitate the large-scale learning process. To support general object rearrangement by physical humanoid, we introduce a novel Human-in-the-Room dataset encompassing various rearrangement tasks. Through extensive experiments and analysis, we demonstrate the effectiveness of the proposed approach.