mlsanityData to 7 Oct 2026

Language-guided Task Adaptation for Imitation Learning

arXiv 2301.09770 ↗First posted 24 Jan 2023cs.AIcited by 2

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

We introduce a novel setting, wherein an agent needs to learn a task from a demonstration of a related task with the difference between the tasks communicated in natural language. The proposed setting allows reusing demonstrations from other tasks, by providing low effort language descriptions, and can also be used to provide feedback to correct agent errors, which are both important desiderata for building intelligent agents that assist humans in daily tasks. To enable progress in this proposed setting, we create two benchmarks -- Room Rearrangement and Room Navigation -- that cover a diverse set of task adaptations. Further, we propose a framework that uses a transformer-based model to reason about the entities in the tasks and their relationships, to learn a policy for the target task

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 34 of its 55 references that are in the map. First posted on arXiv 24 Jan 2023.

Builds on

34 of 55 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020Language-Conditioned Imitation Learning for Robot Manipulation Tasks22 Oct
  2. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation25 Nov

Other references · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2017Graph Attention Networkscited by 8,854
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2016Generative Adversarial Imitation Learningcited by 1,576
  11. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  12. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
20 more
  1. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  2. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  3. 2017A simple neural network module for relational reasoningcited by 345
  4. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  5. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  6. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  7. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  8. 2019A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithmscited by 136
  9. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  10. 2017Learning with Latent Languagecited by 75
  11. 2021Neural Production Systems: Learning Rule-Governed Visual Dynamicscited by 75
  12. 2018Relational Deep Reinforcement Learningcited by 72
  13. 2020Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systemscited by 48
  14. 2021Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learningcited by 45
  15. 2014Learning to Win by Reading Manuals in a Monte-Carlo Frameworkcited by 39
  16. 2020PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewardscited by 30
  17. 2017Beating Atari with Natural Language Guided Reinforcement Learningcited by 24
  18. 2022Policy Architectures for Compositional Generalization in Controlcited by 23
  19. 2015Learning Models for Following Natural Language Directions in Unknown Environmentscited by 8
  20. 2015Robobarista: Object Part based Transfer of Manipulation Trajectories from Crowd-sourcing in 3D Pointcloudscited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1.

2 papers.

  1. 2023FoMo Rewards: Can we cast foundation models as reward functions?cited by 4
  2. 2024SemTra: A Semantic Skill Translator for Cross-Domain Zero-Shot Policy Adaptationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2021Zero-shot Task Adaptation using Natural Language
  2. 2024Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitation
  3. 2023TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Models
  4. 2024Learning with Language-Guided State Abstractions
  5. 2018Guiding Policies with Language via Meta-Learning
  6. 2024Online Adaptation for Enhancing Imitation Learning Policies
  7. 2019Imitation Learning of Robot Policies by Combining Language, Vision and Demonstration
  8. 2026Inner Speech as Behavior Guides: Steerable Imitation of Diverse Behaviors for Human-AI coordination