# LISA: Learning Interpretable Skill Abstractions from Language

arXiv: https://arxiv.org/abs/2203.00054 · HTML: https://mlsanity.com/paper/2203.00054/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 48 of its 59 references that are in the map.
- First posted on arXiv 28 Feb 2022.

## Placed in

- Offline RL and imitation learning from data / [Language-conditioned policies and VLAs](https://mlsanity.com/offlinerl/language-conditioned-policies-and-vlas/)

## Builds on (most cited first, 30 of 48)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [Speaker-Follower Models for Vision-and-Language Navigation](https://mlsanity.com/paper/1806.02724/) (2018)
- [Learning Latent Plans from Play](https://mlsanity.com/paper/1903.01973/) (2019)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout](https://mlsanity.com/paper/1904.04195/) (2019)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [One-Shot Imitation Learning](https://mlsanity.com/paper/1703.07326/) (2017)
- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [One-Shot Visual Imitation Learning via Meta-Learning](https://mlsanity.com/paper/1709.04905/) (2017)
- [Language Conditioned Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2005.07648/) (2020)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Self-Monitoring Navigation Agent via Auxiliary Progress Estimation](https://mlsanity.com/paper/1901.03035/) (2019)
- [A Survey of Reinforcement Learning Informed by Natural Language](https://mlsanity.com/paper/1906.03926/) (2019)
- [One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning](https://mlsanity.com/paper/1802.01557/) (2018)
- [Language as an Abstraction for Hierarchical Deep Reinforcement Learning](https://mlsanity.com/paper/1906.07343/) (2019)

## Built on by (most cited first, 30 of 30)

- [Instruction-driven history-aware policies for robotic manipulations](https://mlsanity.com/paper/2209.04899/) (2022)
- [SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution](https://mlsanity.com/paper/2312.11598/) (2023)
- [Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos](https://mlsanity.com/paper/2412.04445/) (2024)
- [Large Language Models can Implement Policy Iteration](https://mlsanity.com/paper/2210.03821/) (2022)
- [Incremental Learning of Retrievable Skills For Efficient Continual Task Adaptation](https://mlsanity.com/paper/2410.22658/) (2024)
- [Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Tasks](https://mlsanity.com/paper/2210.04476/) (2022)
- [Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidance](https://mlsanity.com/paper/2407.13842/) (2024)
- [STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization](https://mlsanity.com/paper/2506.03863/) (2025)
- [Language-guided Skill Learning with Temporal Variational Inference](https://mlsanity.com/paper/2402.16354/) (2024)
- [Grounding Language Plans in Demonstrations Through Counterfactual Perturbations](https://mlsanity.com/paper/2403.17124/) (2024)
- [Monotonic Location Attention for Length Generalization](https://mlsanity.com/paper/2305.20019/) (2023)
- [Self-driven Grounding: Large Language Model Agents with Automatical Language-aligned Skill Learning](https://mlsanity.com/paper/2309.01352/) (2023)
- [One-shot Imitation in a Non-Stationary Environment via Multi-Modal Skill](https://mlsanity.com/paper/2402.08369/) (2024)
- [DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics](https://mlsanity.com/paper/2312.06408/) (2023)
- [Few-Shot Task Learning through Inverse Generative Modeling](https://mlsanity.com/paper/2411.04987/) (2024)
- [Learning Diffusion Policy from Primitive Skills for Robot Manipulation](https://mlsanity.com/paper/2601.01948/) (2026)
- [A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges](https://mlsanity.com/paper/2211.06665/) (2022)
- [PushWorld: A benchmark for manipulation planning with tools and movable obstacles](https://mlsanity.com/paper/2301.10289/) (2023)
- [Rethinking Mutual Information for Language Conditioned Skill Discovery on Imitation Learning](https://mlsanity.com/paper/2402.17511/) (2024)
- [RoLD: Robot Latent Diffusion for Multi-task Policy Modeling](https://mlsanity.com/paper/2403.07312/) (2024)
- [AC-VLA: Robust Out-of-Distribution Action Execution via Compositional Learning](https://mlsanity.com/paper/2607.15714/) (2026)
- [Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2609.34575/) (2026)
- [SEAL: SEmantic-Augmented Imitation Learning via Language Model](https://mlsanity.com/paper/2410.02231/) (2024)
- [Interpretable Robotic Manipulation from Language](https://mlsanity.com/paper/2405.17047/) (2024)
- [Semantic Skill Grounding for Embodied Instruction-Following in Cross-Domain Environments](https://mlsanity.com/paper/2408.01024/) (2024)
- [CROPS: A Deployable Crop Management System Over All Possible State Availabilities](https://mlsanity.com/paper/2411.06034/) (2024)
- [EVLP:Learning Unified Embodied Vision-Language Planner with Reinforced Supervised Fine-Tuning](https://mlsanity.com/paper/2511.05553/) (2025)
- [The Value of Mechanistic Priors in Sequential Decision Making](https://mlsanity.com/paper/2605.10018/) (2026)
- [BooST: Bridging Semantics and Motions for Efficient Skill Transfer](https://mlsanity.com/paper/2608.10600/) (2026)
- [Decoupling Planning and Control for Instructable Agents](https://mlsanity.com/paper/2608.26788/) (2026)

## Abstract (arXiv)

Learning policies that effectively utilize language instructions in complex, multi-task environments is an important problem in sequential decision-making. While it is possible to condition on the entire language instruction directly, such an approach could suffer from generalization issues. In our work, we propose Learning Interpretable Skill Abstractions (LISA), a hierarchical imitation learning framework that can learn diverse, interpretable primitive behaviors or skills from language-conditioned demonstrations to better generalize to unseen instructions. LISA uses vector quantization to learn discrete skill codes that are highly correlated with language instructions and the behavior of the learned policy. In navigation and robotic manipulation environments, LISA outperforms a strong non-hierarchical Decision Transformer baseline in the low data regime and is able to compose learned skills to solve tasks containing unseen long-range instructions. Our method demonstrates a more natural way to condition on language in sequential decision-making problems and achieve interpretable and controllable behavior with the learned skills.