# Adaptive Language-Guided Abstraction from Contrastive Explanations

arXiv: https://arxiv.org/abs/2409.08212 · HTML: https://mlsanity.com/paper/2409.08212/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 12 Sep 2024.

## Builds on (most cited first, 21 of 21)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning](https://mlsanity.com/paper/2310.12921/) (2023)
- [AI Safety Gridworlds](https://mlsanity.com/paper/1711.09883/) (2017)
- [Correcting Robot Plans with Natural Language Feedback](https://mlsanity.com/paper/2204.05186/) (2022)
- [Feature Expansive Reward Learning: Rethinking Human Input](https://mlsanity.com/paper/2006.13208/) (2020)
- [Practice Makes Perfect: Planning to Learn Skill Parameter Policies](https://mlsanity.com/paper/2402.15025/) (2024)
- [How to talk so AI will learn: Instructions, descriptions, and autonomy](https://mlsanity.com/paper/2206.07870/) (2022)
- [Inducing Structure in Reward Learning by Learning Features](https://mlsanity.com/paper/2201.07082/) (2022)
- [Pragmatic inference and visual abstraction enable contextual flexibility during visual communication](https://mlsanity.com/paper/1903.04448/) (2019)
- [Preference-Conditioned Language-Guided Abstraction](https://mlsanity.com/paper/2402.03081/) (2024)
- [LaMPP: Language Models as Probabilistic Priors for Perception and Action](https://mlsanity.com/paper/2302.02801/) (2023)
- [Learning with Language-Guided State Abstractions](https://mlsanity.com/paper/2402.18759/) (2024)

## Built on by (most cited first, 8 of 8)

- [Pure Vision Language Action (VLA) Models: A Comprehensive Survey](https://mlsanity.com/paper/2509.19012/) (2025)
- [ELEMENTAL: Interactive Learning from Demonstrations and Vision-Language Models for Reward Design in Robotics](https://mlsanity.com/paper/2411.18825/) (2024)
- [Masked IRL: LLM-Guided Reward Disambiguation from Demonstrations and Language](https://mlsanity.com/paper/2511.14565/) (2025)
- [QuickLAP: Quick Language-Action Preference Learning for Semi-Autonomous Agents](https://mlsanity.com/paper/2511.17855/) (2025)
- [Learning Transferable Latent User Preferences for Human-Aligned Decision Making](https://mlsanity.com/paper/2605.12682/) (2026)
- [Robots That Know What to Ask: Recovering Misaligned Rewards through Targeted Explanations](https://mlsanity.com/paper/2605.22986/) (2026)
- [Bayesian Active Learning for Intent Disambiguation in Interactive Robot Planning](https://mlsanity.com/paper/2609.34270/) (2026)
- [Iterative Policy Refinement through Semantic Rollout Analysis](https://mlsanity.com/paper/2610.01652/) (2026)

## Abstract (arXiv)

Many approaches to robot learning begin by inferring a reward function from a set of human demonstrations. To learn a good reward, it is necessary to determine which features of the environment are relevant before determining how these features should be used to compute reward. End-to-end methods for joint feature and reward learning (e.g., using deep networks or program synthesis techniques) often yield brittle reward functions that are sensitive to spurious state features. By contrast, humans can often generalizably learn from a small number of demonstrations by incorporating strong priors about what features of a demonstration are likely meaningful for a task of interest. How do we build robots that leverage this kind of background knowledge when learning from new demonstrations? This paper describes a method named ALGAE (Adaptive Language-Guided Abstraction from [Contrastive] Explanations) which alternates between using language models to iteratively identify human-meaningful features needed to explain demonstrated behavior, then standard inverse reinforcement learning techniques to assign weights to these features. Experiments across a variety of both simulated and real-world robot environments show that ALGAE learns generalizable reward functions defined on interpretable features using only small numbers of demonstrations. Importantly, ALGAE can recognize when features are missing, then extract and define those features without any human input -- making it possible to quickly and efficiently acquire rich representations of user behavior.