# DASH: Modularized Human Manipulation Simulation with Vision and Language for Embodied AI

arXiv: https://arxiv.org/abs/2108.12536 · HTML: https://mlsanity.com/paper/2108.12536/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 93 references that are in the map.
- First posted on arXiv 28 Aug 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [Deep Dynamics Models for Learning Dexterous Manipulation](https://mlsanity.com/paper/1909.11652/) (2019)
- [Deep Convolutional Inverse Graphics Network](https://mlsanity.com/paper/1503.03167/) (2015)
- [State-Only Imitation Learning for Dexterous Manipulation](https://mlsanity.com/paper/2004.04650/) (2020)
- [Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environments](https://mlsanity.com/paper/1910.14442/) (2019)
- [Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions](https://mlsanity.com/paper/1710.06280/) (2017)
- [Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction](https://mlsanity.com/paper/1806.03831/) (2018)
- [Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations](https://mlsanity.com/paper/1603.04351/) (2016)
- [AllenAct: A Framework for Embodied AI Research](https://mlsanity.com/paper/2008.12760/) (2020)
- [Assistive Gym: A Physics Simulation Framework for Assistive Robotics](https://mlsanity.com/paper/1910.04700/) (2019)
- [From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood](https://mlsanity.com/paper/1704.07926/) (2017)
- [Prospection: Interpretable Plans From Language By Predicting the Future](https://mlsanity.com/paper/1903.08309/) (2019)
- [SFV: Reinforcement Learning of Physical Skills from Videos](https://mlsanity.com/paper/1810.03599/) (2018)
- [Learning Object Placements For Relational Instructions by Hallucinating Scene Representations](https://mlsanity.com/paper/2001.08481/) (2020)
- [Question Asking as Program Generation](https://mlsanity.com/paper/1711.06351/) (2017)
- [Leveraging Contact Forces for Learning to Grasp](https://mlsanity.com/paper/1809.07004/) (2018)
- [A Review of Methodologies for Natural-Language-Facilitated Human-Robot Cooperation](https://mlsanity.com/paper/1701.08756/) (2017)
- [Catch & Carry: Reusable Neural Controllers for Vision-Guided Whole-Body Tasks](https://mlsanity.com/paper/1911.06636/) (2019)

## Built on by (most cited first, 2 of 2)

- [D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions](https://mlsanity.com/paper/2112.03028/) (2021)
- [Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learning](https://mlsanity.com/paper/2505.12278/) (2025)

## Abstract (arXiv)

Creating virtual humans with embodied, human-like perceptual and actuation constraints has the promise to provide an integrated simulation platform for many scientific and engineering applications. We present Dynamic and Autonomous Simulated Human (DASH), an embodied virtual human that, given natural language commands, performs grasp-and-stack tasks in a physically-simulated cluttered environment solely using its own visual perception, proprioception, and touch, without requiring human motion data. By factoring the DASH system into a vision module, a language module, and manipulation modules of two skill categories, we can mix and match analytical and machine learning techniques for different modules so that DASH is able to not only perform randomly arranged tasks with a high success rate, but also do so under anthropomorphic constraints and with fluid and diverse motions. The modular design also favors analysis and extensibility to more complex manipulation skills.