# Learning to Learn Faster from Human Feedback with Language Model Predictive Control

arXiv: https://arxiv.org/abs/2402.11450 · HTML: https://mlsanity.com/paper/2402.11450/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2024.

## Built on by (most cited first, 26 of 26)

- [Vision Language Models are In-Context Value Learners](https://mlsanity.com/paper/2411.04549/) (2024)
- [VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought](https://mlsanity.com/paper/2406.14596/) (2024)
- [Expanding the Capabilities of Reinforcement Learning via Text Feedback](https://mlsanity.com/paper/2602.02482/) (2026)
- [CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Models](https://mlsanity.com/paper/2409.18382/) (2024)
- [AIC MLLM: Autonomous Interactive Correction MLLM for Robust Robotic Manipulation](https://mlsanity.com/paper/2406.11548/) (2024)
- [Automating Robot Failure Recovery Using Vision-Language Models With Optimized Prompts](https://mlsanity.com/paper/2409.03966/) (2024)
- [Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models](https://mlsanity.com/paper/2407.02666/) (2024)
- [On Almost Surely Safe Alignment of Large Language Models at Inference-Time](https://mlsanity.com/paper/2502.01208/) (2025)
- [M3HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality](https://mlsanity.com/paper/2503.02077/) (2025)
- [Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Models](https://mlsanity.com/paper/2504.13351/) (2025)
- [Growing with Your Embodied Agent: A Human-in-the-Loop Lifelong Code Generation Framework for Long-Horizon Manipulation Skills](https://mlsanity.com/paper/2509.18597/) (2025)
- [FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models](https://mlsanity.com/paper/2407.11522/) (2024)
- [ADAPT: Actively Discovering and Adapting to Preferences for any Task](https://mlsanity.com/paper/2504.04040/) (2025)
- [A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewards](https://mlsanity.com/paper/2502.08643/) (2025)
- [EmboMatrix: A Scalable Training-Ground for Embodied Decision-Making](https://mlsanity.com/paper/2510.12072/) (2025)
- [A Principle of Targeted Intervention for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2510.17697/) (2025)
- [GRAPPA: Generalizing and Adapting Robot Policies via Online Agentic Guidance](https://mlsanity.com/paper/2410.06473/) (2024)
- [Robot Behavior Personalization from Sparse User Feedback](https://mlsanity.com/paper/2410.19219/) (2024)
- [Thinking Forward and Backward: Effective Backward Planning with Large Language Models](https://mlsanity.com/paper/2411.01790/) (2024)
- [Game On: Towards Language Models as RL Experimenters](https://mlsanity.com/paper/2409.03402/) (2024)
- [Efficiently Generating Expressive Quadruped Behaviors via Language-Guided Preference Learning](https://mlsanity.com/paper/2502.03717/) (2025)
- [Unfettered Forceful Skill Acquisition with Physical Reasoning and Coordinate Frame Labeling](https://mlsanity.com/paper/2505.09731/) (2025)
- [See it to Place it: Evolving Macro Placements with Vision-Language Models](https://mlsanity.com/paper/2603.28733/) (2026)
- [In-Context Learning for Robots: Methods and Applications](https://mlsanity.com/paper/2609.36012/) (2026)
- [Language-Guided Terrain-Adaptive Neural MPC for Autonomous Traversal of Articulated Tracked Robots](https://mlsanity.com/paper/2609.13083/) (2026)
- [InterEvolve: Test-Time Evolution of Reward Programs for Humanoid Loco-Manipulation](https://mlsanity.com/paper/2610.02196/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have been shown to exhibit a wide range of capabilities, such as writing robot code from language commands -- enabling non-experts to direct robot behaviors, modify them based on feedback, or compose them to perform new tasks. However, these capabilities (driven by in-context learning) are limited to short-term interactions, where users' feedback remains relevant for only as long as it fits within the context size of the LLM, and can be forgotten over longer interactions. In this work, we investigate fine-tuning the robot code-writing LLMs, to remember their in-context interactions and improve their teachability i.e., how efficiently they adapt to human inputs (measured by average number of corrections before the user considers the task successful). Our key observation is that when human-robot interactions are viewed as a partially observable Markov decision process (in which human language inputs are observations, and robot code outputs are actions), then training an LLM to complete previous interactions is training a transition dynamics model -- that can be combined with classic robotics techniques such as model predictive control (MPC) to discover shorter paths to success. This gives rise to Language Model Predictive Control (LMPC), a framework that fine-tunes PaLM 2 to improve its teachability on 78 tasks across 5 robot embodiments -- improving non-expert teaching success rates of unseen tasks by 26.9% while reducing the average number of human corrections from 2.4 to 1.9. Experiments show that LMPC also produces strong meta-learners, improving the success rate of in-context learning new tasks on unseen robot embodiments and APIs by 31.5%. See videos, code, and demos at: https://robot-teaching.github.io/.