mlsanityData to 7 Oct 2026

LISA: Learning Interpretable Skill Abstractions from Language

arXiv 2203.00054 ↗First posted 28 Feb 2022cs.LG cs.AI cs.CLcited by 30

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

Learning policies that effectively utilize language instructions in complex, multi-task environments is an important problem in sequential decision-making. While it is possible to condition on the entire language instruction directly, such an approach could suffer from generalization issues. In our work, we propose Learning Interpretable Skill Abstractions (LISA), a hierarchical imitation learning framework that can learn diverse, interpretable primitive behaviors or skills from language-conditioned demonstrations to better generalize to unseen instructions. LISA uses vector quantization to learn discrete skill codes that are highly correlated with language instructions and the behavior of the learned policy. In navigation and robotic manipulation environments, LISA outperforms a strong non-hierarchical Decision Transformer baseline in the low data regime and is able to compose learned skills to solve tasks containing unseen long-range instructions. Our method demonstrates a more natural way to condition on language in sequential decision-making problems and achieve interpretable and controllable behavior with the learned skills.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 48 of its 59 references that are in the map. First posted on arXiv 28 Feb 2022.

Builds on

48 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020Language-Conditioned Imitation Learning for Robot Manipulation Tasks22 Oct
  2. 2020Language Conditioned Imitation Learning over Unstructured Data15 May

Other references · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  8. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  9. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  10. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  11. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  12. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
34 more
  1. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  2. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  3. 2019Learning Latent Plans from Playcited by 310
  4. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  5. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  6. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  7. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  8. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  9. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  10. 2017One-Shot Imitation Learningcited by 232
  11. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  12. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  13. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  14. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  15. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  16. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  17. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  18. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  19. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  20. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  21. 2020A unifying mutual information view of metric learning: cross-entropy vs. pairwise lossescited by 100
  22. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 97
  23. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  24. 2017Neural Task Programming: Learning to Generalize Across Hierarchical Taskscited by 75
  25. 2017Gated-Attention Architectures for Task-Oriented Language Groundingcited by 72
  26. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 57
  27. 2019Sub-policy Adaptation for Hierarchical Reinforcement Learningcited by 47
  28. 2016Learning Language Games through Interactioncited by 43
  29. 2020Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learningcited by 32
  30. 2019Hierarchical Decision Making by Generating and Following Natural Language Instructionscited by 30
  31. 2018Following High-level Navigation Instructions on a Simulated Quadcopter with Imitation Learningcited by 27
  32. 2019Prospection: Interpretable Plans From Language By Predicting the Futurecited by 21
  33. 2017Accurately and Efficiently Interpreting Human-Robot Instructions of Varying Granularitiescited by 19
  34. 2018Situated Mapping of Sequential Instructions to Actions with Single-step Reward Observationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 5; 2024: 13; 2025: 2; 2026: 6.

30 papers.

Sort:

By area:

  1. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  2. 2023SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Executioncited by 69
  3. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 65
  4. 2022Large Language Models can Implement Policy Iterationcited by 31
  5. 2024Incremental Learning of Retrievable Skills For Efficient Continual Task Adaptationcited by 19
  6. 2022Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Taskscited by 17
  7. 2024Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidancecited by 16
  8. 2025STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantizationcited by 13
  9. 2024Language-guided Skill Learning with Temporal Variational Inferencecited by 11
  10. 2024Grounding Language Plans in Demonstrations Through Counterfactual Perturbationscited by 11
  11. 2023Monotonic Location Attention for Length Generalizationcited by 9
  12. 2023Self-driven Grounding: Large Language Model Agents with Automatical Language-aligned Skill Learningcited by 9
18 more
  1. 2024One-shot Imitation in a Non-Stationary Environment via Multi-Modal Skillcited by 7
  2. 2023DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physicscited by 6
  3. 2024Few-Shot Task Learning through Inverse Generative Modelingcited by 4
  4. 2022A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challengescited by 3
  5. 2023PushWorld: A benchmark for manipulation planning with tools and movable obstaclescited by 3
  6. 2024Rethinking Mutual Information for Language Conditioned Skill Discovery on Imitation Learningcited by 3
  7. 2024RoLD: Robot Latent Diffusion for Multi-task Policy Modelingcited by 3
  8. 2026Learning Diffusion Policy from Primitive Skills for Robot Manipulationcited by 3
  9. 2026AC-VLA: Robust Out-of-Distribution Action Execution via Compositional Learningcited by 3
  10. 2024SEAL: SEmantic-Augmented Imitation Learning via Language Modelcited by 1
  11. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learningcited by 1
  12. 2024Interpretable Robotic Manipulation from Languagecited by 0
  13. 2024Semantic Skill Grounding for Embodied Instruction-Following in Cross-Domain Environmentscited by 0
  14. 2024CROPS: A Deployable Crop Management System Over All Possible State Availabilitiescited by 0
  15. 2025EVLP:Learning Unified Embodied Vision-Language Planner with Reinforced Supervised Fine-Tuningcited by 0
  16. 2026The Value of Mechanistic Priors in Sequential Decision Makingcited by 0
  17. 2026BooST: Bridging Semantics and Motions for Efficient Skill Transfercited by 0
  18. 2026Decoupling Planning and Control for Instructable Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Dual-Process Atomic Skill Learning: Decoupling Semantic Reasoning and Real-Time Control
  2. 2025CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguity
  3. 2025From Code to Action: Hierarchical Learning of Diffusion-VLM Policies
  4. 2024Learning with Language-Guided State Abstractions
  5. 2024Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitation
  6. 2026Multi-Granularity Language-Guided Imitation Learning via Instruction Decomposition
  7. 2024Learning Manipulation Skills through Robot Chain-of-Thought with Sparse Failure Guidance
  8. 2025Learning Compositional Behaviors from Demonstration and Language