mlsanityData to 14 Oct 2026

Zero-shot Task Adaptation using Natural Language

arXiv 2106.02972 ↗First posted 5 Jun 2021cs.AI cs.CL cs.LGcited by 15

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

Imitation learning and instruction-following are two common approaches to communicate a user's intent to a learning agent. However, as the complexity of tasks grows, it could be beneficial to use both demonstrations and language to communicate with an agent. In this work, we propose a novel setting where an agent is given both a demonstration and a description, and must combine information from both the modalities. Specifically, given a demonstration for a task (the source task), and a natural language description of the differences between the demonstrated task and a related but different task (the target task), our goal is to train an agent to complete the target task in a zero-shot setting, that is, without any demonstrations for the target task. To this end, we introduce Language-Aided Reward and Value Adaptation (LARVA) which, given a source demonstration and a linguistic description of how the target task differs, learns to output a reward / value function that accurately describes the target task. Our experiments show that on a diverse set of adaptations, our approach is able to complete more than 95% of target tasks when using template-based descriptions, and more than 70% when using free-form natural language.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 39 references that are in the map. First posted on arXiv 5 Jun 2021.

Builds on

21 of 39 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020Language-Conditioned Imitation Learning for Robot Manipulation Tasks22 Oct
  2. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation25 Nov

Other references · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Generative Adversarial Imitation Learningcited by 1,597
  3. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  4. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  5. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
  6. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  7. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  8. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  9. 2018Meta-Learning: A Surveycited by 256
  10. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  11. 2019A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithmscited by 142
  12. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
7 more
  1. 2017Learning with Latent Languagecited by 75
  2. 2021Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learningcited by 45
  3. 2014Learning to Win by Reading Manuals in a Monte-Carlo Frameworkcited by 39
  4. 2020PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewardscited by 30
  5. 2017Beating Atari with Natural Language Guided Reinforcement Learningcited by 24
  6. 2015Learning Models for Following Natural Language Directions in Unknown Environmentscited by 8
  7. 2015Robobarista: Object Part based Transfer of Manipulation Trajectories from Crowd-sourcing in 3D Pointcloudscited by 3

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 6; 2024: 2; 2025: 2.

15 papers.

Sort:

By area:

  1. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  2. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
  3. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
  4. 2022LATTE: LAnguage Trajectory TransformErcited by 40
  5. 2023KITE: Keypoint-Conditioned Policies for Semantic Manipulationcited by 39
  6. 2022Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformerscited by 17
  7. 2024Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networkscited by 14
  8. 2023Improving Long-Horizon Imitation Through Instruction Predictioncited by 8
  9. 2024RLZero: Direct Policy Inference from Language Without In-Domain Supervisioncited by 7
  10. 2023SAGE: Bridging Semantic and Actionable Parts for GEneralizable Manipulation of Articulated Objectscited by 5
  11. 2022Compositional Generalization in Grounded Language Learning via Induced Model Sparsitycited by 4
  12. 2023FoMo Rewards: Can we cast foundation models as reward functions?cited by 4
3 more
  1. 2023Improved Compositional Generalization by Generating Demonstrations for Meta-Learningcited by 3
  2. 2025Bridging Perception and Action: Spatially-Grounded Mid-Level Representations for Robot Generalizationcited by 3
  3. 2025A Human-in-the-loop Approach to Robot Action Replanning through LLM Common-Sense Reasoningcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2023Language-guided Task Adaptation for Imitation Learning
  2. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
  3. 2018Guiding Policies with Language via Meta-Learning
  4. 2020Language Conditioned Imitation Learning over Unstructured Data
  5. 2023Language-Conditioned Imitation Learning with Base Skill Priors under Unstructured Data
  6. 2026Language-Critique Imitation Learning from Suboptimal Demonstrations
  7. 2025Masked IRL: LLM-Guided Reward Disambiguation from Demonstrations and Language
  8. 2026Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learning