# Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning

arXiv: https://arxiv.org/abs/2302.02662 · HTML: https://mlsanity.com/paper/2302.02662/

## Key facts

- Cited by 195 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 63 references that are in the map.
- First posted on arXiv 6 Feb 2023.

## Builds on (most cited first, 30 of 46)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks](https://mlsanity.com/paper/2206.08916/) (2022)
- [VIMA: General Robot Manipulation with Multimodal Prompts](https://mlsanity.com/paper/2210.03094/) (2022)
- [ScienceWorld: Is your Agent Smarter than a 5th Grader?](https://mlsanity.com/paper/2203.07540/) (2022)
- [TextWorld: A Learning Environment for Text-based Games](https://mlsanity.com/paper/1806.11532/) (2018)
- [Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://mlsanity.com/paper/2210.01241/) (2022)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [Experience Grounds Language](https://mlsanity.com/paper/2004.10151/) (2020)
- [Pre-Trained Language Models for Interactive Decision-Making](https://mlsanity.com/paper/2202.01771/) (2022)

## Built on by (most cited first, 30 of 195)

- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Visual-RFT: Visual Reinforcement Fine-Tuning](https://mlsanity.com/paper/2503.01785/) (2025)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives](https://mlsanity.com/paper/2312.11970/) (2023)
- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [Text2Reward: Reward Shaping with Language Models for Reinforcement Learning](https://mlsanity.com/paper/2309.11489/) (2023)
- [Teaching Large Language Models to Reason with Reinforcement Learning](https://mlsanity.com/paper/2403.04642/) (2024)
- [Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning](https://mlsanity.com/paper/2405.10292/) (2024)
- [ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy](https://mlsanity.com/paper/2502.05450/) (2025)
- [RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning](https://mlsanity.com/paper/2410.02089/) (2024)
- [BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games](https://mlsanity.com/paper/2411.13543/) (2024)
- [DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning](https://mlsanity.com/paper/2402.17453/) (2024)
- [Language Models Meet World Models: Embodied Experiences Enhance Language Models](https://mlsanity.com/paper/2305.10626/) (2023)
- [Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods](https://mlsanity.com/paper/2404.00282/) (2024)
- [Improving Vision-Language-Action Model with Online Reinforcement Learning](https://mlsanity.com/paper/2501.16664/) (2025)
- [A Survey on Large Language Model-Based Game Agents](https://mlsanity.com/paper/2404.02039/) (2024)
- [Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis](https://mlsanity.com/paper/2312.08782/) (2023)
- [OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models](https://mlsanity.com/paper/2410.09671/) (2024)
- [Large Language Models as Generalizable Policies for Embodied Tasks](https://mlsanity.com/paper/2310.17722/) (2023)
- [o1-Coder: an o1 Replication for Coding](https://mlsanity.com/paper/2412.00154/) (2024)
- [Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generation](https://mlsanity.com/paper/2311.04254/) (2023)
- [SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience](https://mlsanity.com/paper/2508.04700/) (2025)
- [Learning to Model the World with Language](https://mlsanity.com/paper/2308.01399/) (2023)
- [Large Language Models in Education: Vision and Opportunities](https://mlsanity.com/paper/2311.13160/) (2023)
- [SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors](https://mlsanity.com/paper/2403.13438/) (2024)
- [Large Generative AI Models for Telecom: The Next Big Thing?](https://mlsanity.com/paper/2306.10249/) (2023)
- [Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models](https://mlsanity.com/paper/2503.02623/) (2025)
- [MARFT: Multi-Agent Reinforcement Fine-Tuning](https://mlsanity.com/paper/2504.16129/) (2025)
- [Tree-Planner: Efficient Close-loop Task Planning with Large Language Models](https://mlsanity.com/paper/2310.08582/) (2023)

## Abstract (arXiv)

Recent works successfully leveraged Large Language Models' (LLM) abilities to capture abstract knowledge about world's physics to solve decision-making problems. Yet, the alignment between LLMs' knowledge and the environment can be wrong and limit functional competence due to lack of grounding. In this paper, we study an approach (named GLAM) to achieve this alignment through functional grounding: we consider an agent using an LLM as a policy that is progressively updated as the agent interacts with the environment, leveraging online Reinforcement Learning to improve its performance to solve goals. Using an interactive textual environment designed to study higher-level forms of functional grounding, and a set of spatial and navigation tasks, we study several scientific questions: 1) Can LLMs boost sample efficiency for online learning of various RL tasks? 2) How can it boost different forms of generalization? 3) What is the impact of online learning? We study these questions by functionally grounding several variants (size, architecture) of FLAN-T5.