# ASC me to Do Anything: Multi-task Training for Embodied AI

arXiv: https://arxiv.org/abs/2202.06987 · HTML: https://mlsanity.com/paper/2202.06987/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 78 references that are in the map.
- First posted on arXiv 14 Feb 2022.

## Builds on (most cited first, 30 of 67)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [Embodied Question Answering](https://mlsanity.com/paper/1711.11543/) (2017)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [12-in-1: Multi-Task Vision and Language Representation Learning](https://mlsanity.com/paper/1912.02315/) (2019)
- [ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation](https://mlsanity.com/paper/2007.04954/) (2020)
- [ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects](https://mlsanity.com/paper/2006.13171/) (2020)
- [The Natural Language Decathlon: Multitask Learning as Question Answering](https://mlsanity.com/paper/1806.08730/) (2018)

## Built on by (most cited first, 1 of 1)

- [AI-Driven Relocation Tracking in Dynamic Kitchen Environments](https://mlsanity.com/paper/2503.01547/) (2025)

## Abstract (arXiv)

Embodied AI has seen steady progress across a diverse set of independent tasks. While these varied tasks have different end goals, the basic skills required to complete them successfully overlap significantly. In this paper, our goal is to leverage these shared skills to learn to perform multiple tasks jointly. We propose Atomic Skill Completion (ASC), an approach for multi-task training for Embodied AI, where a set of atomic skills shared across multiple tasks are composed together to perform the tasks. The key to the success of this approach is a pre-training scheme that decouples learning of the skills from the high-level tasks making joint training effective. We use ASC to train agents within the AI2-THOR environment to perform four interactive tasks jointly and find it to be remarkably effective. In a multi-task setting, ASC improves success rates by a factor of 2x on Seen scenes and 4x on Unseen scenes compared to no pre-training. Importantly, ASC enables us to train a multi-task agent that has a 52% higher Success Rate than training 4 independent single task agents. Finally, our hierarchical agents are more interpretable than traditional black-box architectures.