# Language Instructed Reinforcement Learning for Human-AI Coordination

arXiv: https://arxiv.org/abs/2304.07297 · HTML: https://mlsanity.com/paper/2304.07297/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 13 Apr 2023.

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- ["Other-Play" for Zero-Shot Coordination](https://mlsanity.com/paper/2003.02979/) (2020)
- [The Hanabi Challenge: A New Frontier for AI Research](https://mlsanity.com/paper/1902.00506/) (2019)
- [Equivalence Between Policy Gradients and Soft Q-Learning](https://mlsanity.com/paper/1704.06440/) (2017)
- [Collaborating with Humans without Human Data](https://mlsanity.com/paper/2110.08176/) (2021)
- [Off-Belief Learning](https://mlsanity.com/paper/2103.04000/) (2021)
- [Semantic Exploration from Language Abstractions and Pretrained Representations](https://mlsanity.com/paper/2204.05080/) (2022)
- [Improving Intrinsic Exploration with Language Abstractions](https://mlsanity.com/paper/2202.08938/) (2022)
- [Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning](https://mlsanity.com/paper/2210.05492/) (2022)
- [ELLA: Exploration through Learned Language Abstraction](https://mlsanity.com/paper/2103.05825/) (2021)
- [K-level Reasoning for Zero-Shot Coordination in Hanabi](https://mlsanity.com/paper/2207.07166/) (2022)
- [The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Models](https://mlsanity.com/paper/2204.10759/) (2022)
- [On the Critical Role of Conventions in Adaptive Human-AI Collaboration](https://mlsanity.com/paper/2104.02871/) (2021)
- [Using Natural Language and Program Abstractions to Instill Human Inductive Biases in Machines](https://mlsanity.com/paper/2205.11558/) (2022)

## Built on by (most cited first, 30 of 63)

- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [Large Language Models as General Pattern Machines](https://mlsanity.com/paper/2307.04721/) (2023)
- [LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving](https://mlsanity.com/paper/2310.03026/) (2023)
- [RoboCLIP: One Demonstration is Enough to Learn Robot Policies](https://mlsanity.com/paper/2310.07899/) (2023)
- [Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods](https://mlsanity.com/paper/2404.00282/) (2024)
- [Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis](https://mlsanity.com/paper/2312.08782/) (2023)
- [Large Language Models for Robotics: Opportunities, Challenges, and Perspectives](https://mlsanity.com/paper/2401.04334/) (2024)
- [Large Language Models as Generalizable Policies for Embodied Tasks](https://mlsanity.com/paper/2310.17722/) (2023)
- [Vision-Language Models as a Source of Rewards](https://mlsanity.com/paper/2312.09187/) (2023)
- [Imitation Bootstrapped Reinforcement Learning](https://mlsanity.com/paper/2311.02198/) (2023)
- [Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Corrections](https://mlsanity.com/paper/2311.10678/) (2023)
- [Self-Refined Large Language Model as Automated Reward Function Designer for Deep Reinforcement Learning in Robotics](https://mlsanity.com/paper/2309.06687/) (2023)
- [True Knowledge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learning](https://mlsanity.com/paper/2401.14151/) (2024)
- [PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learning](https://mlsanity.com/paper/2405.02501/) (2024)
- [Can ChatGPT Enable ITS? The Case of Mixed Traffic Control via Reinforcement Learning](https://mlsanity.com/paper/2306.08094/) (2023)
- [The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Models](https://mlsanity.com/paper/2402.01874/) (2024)
- [Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Tasks](https://mlsanity.com/paper/2604.20987/) (2026)
- [How Can LLM Guide RL? A Value-Based Approach](https://mlsanity.com/paper/2402.16181/) (2024)
- [Asking Before Acting: Gather Information in Embodied Decision Making with Language Models](https://mlsanity.com/paper/2305.15695/) (2023)
- [Multi-Agent, Human-Agent and Beyond: A Survey on Cooperation in Social Dilemmas](https://mlsanity.com/paper/2402.17270/) (2024)
- [M3HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality](https://mlsanity.com/paper/2503.02077/) (2025)
- [Guide Your Agent with Adaptive Multimodal Rewards](https://mlsanity.com/paper/2309.10790/) (2023)
- [Towards Collaborative Intelligence: Propagating Intentions and Reasoning for Multi-Agent Coordination with Large Language Models](https://mlsanity.com/paper/2407.12532/) (2024)
- [The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning](https://mlsanity.com/paper/2502.15214/) (2025)
- [Toward Grounded Commonsense Reasoning](https://mlsanity.com/paper/2306.08651/) (2023)
- [A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedback](https://mlsanity.com/paper/2411.13410/) (2024)
- [Video2Reward: Generating Reward Function from Videos for Legged Robot Behavior Learning](https://mlsanity.com/paper/2412.05515/) (2024)
- [TeViR: Text-to-Video Reward with Diffusion Models for Efficient Reinforcement Learning](https://mlsanity.com/paper/2505.19769/) (2025)
- [FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making](https://mlsanity.com/paper/2507.12496/) (2025)

## Abstract (arXiv)

One of the fundamental quests of AI is to produce agents that coordinate well with humans. This problem is challenging, especially in domains that lack high quality human behavioral data, because multi-agent reinforcement learning (RL) often converges to different equilibria from the ones that humans prefer. We propose a novel framework, instructRL, that enables humans to specify what kind of strategies they expect from their AI partners through natural language instructions. We use pretrained large language models to generate a prior policy conditioned on the human instruction and use the prior to regularize the RL objective. This leads to the RL agent converging to equilibria that are aligned with human preferences. We show that instructRL converges to human-like policies that satisfy the given instructions in a proof-of-concept environment as well as the challenging Hanabi benchmark. Finally, we show that knowing the language instruction significantly boosts human-AI coordination performance in human evaluations in Hanabi.