ASC me to Do Anything: Multi-task Training for Embodied AI
Not on the map. Why?
Abstract (arXiv)
Embodied AI has seen steady progress across a diverse set of independent tasks. While these varied tasks have different end goals, the basic skills required to complete them successfully overlap significantly. In this paper, our goal is to leverage these shared skills to learn to perform multiple tasks jointly. We propose Atomic Skill Completion (ASC), an approach for multi-task training for Embodied AI, where a set of atomic skills shared across multiple tasks are composed together to perform the tasks. The key to the success of this approach is a pre-training scheme that decouples learning of the skills from the high-level tasks making joint training effective. We use ASC to train agents within the AI2-THOR environment to perform four interactive tasks jointly and find it to be remarkably effective. In a multi-task setting, ASC improves success rates by a factor of 2x on Seen scenes and 4x on Unseen scenes compared to no pre-training. Importantly, ASC enables us to train a multi-task agent that has a 52% higher Success Rate than training 4 independent single task agents. Finally, our hierarchical agents are more interpretable than traditional black-box architectures.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 67 of its 78 references that are in the map. First posted on arXiv 14 Feb 2022.
Builds on
67 of 78 references.
References · 67
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2018Deep contextualized word representationscited by 2,290
- 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
- 2019Objects as Pointscited by 1,175
- 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
- 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
- 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
- 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
38 more
- 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
- 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
- 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
- 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
- 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
- 2018On Evaluation of Embodied Navigation Agentscited by 547
- 2018End-to-End Multi-Task Learning with Attentioncited by 534
- 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
- 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
- 2016Cross-stitch Networks for Multi-task Learningcited by 467
- 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
- 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
- 2017Embodied Question Answeringcited by 336
- 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
- 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
- 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 250
- 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 244
- 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 236
- 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 203
- 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
- 2021Episodic Transformer for Vision-and-Language Navigationcited by 156
- 2017IQA: Visual Question Answering in Interactive Environmentscited by 155
- 2020Rearrangement: A Challenge for Embodied AIcited by 155
- 2020iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenescited by 155
- 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
- 2019Large-scale weakly-supervised pre-training for video action recognitioncited by 130
- 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
- 2021Visual Room Rearrangementcited by 102
- 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 102
- 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 98
- 2018Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learningcited by 93
- 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
- 2018Hierarchical Imitation and Reinforcement Learningcited by 88
- 2017One Model To Learn Them Allcited by 87
- 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
- 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
- 2017Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learningcited by 77
- 2017Cross-Domain Self-supervised Multi-task Feature Learning using Synthetic Imagerycited by 56
50 most cited of 67 All 67 (JSON)
Built on by
Citing papers per year: 2025: 1.
1 paper.