mlsanityData to 7 Oct 2026

LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models

arXiv 2311.18232 ↗First posted 30 Nov 2023cs.CL cs.AI cs.LGcited by 84

Not on the map. Why?

Cited by 36 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) provide excellent text-generation capabilities, but standard prompting and generation methods generally do not lead to intentional or goal-directed agents and might necessitate considerable prompt tuning. This becomes particularly apparent in multi-turn conversations: even the best current LLMs rarely ask clarifying questions, engage in explicit information gathering, or take actions now that lead to better decisions after multiple turns. Reinforcement learning has the potential to leverage the powerful modeling capabilities of LLMs, as well as their internal representation of textual interactions, to create capable goal-directed language agents. This can enable intentional and temporally extended interactions, such as with humans, through coordinated persuasion and carefully crafted questions, or in goal-directed play through text games to bring about desired final outcomes. However, enabling this requires the community to develop stable and reliable reinforcement learning algorithms that can effectively train LLMs. Developing such algorithms requires tasks that can gauge progress on algorithm design, provide accessible and reproducible evaluations for multi-turn interactions, and cover a range of task properties and challenges in improving reinforcement learning algorithms. Our paper introduces the LMRL-Gym benchmark for evaluating multi-turn RL for LLMs, together with an open-source research framework containing a basic toolkit for getting started on multi-turn RL with offline value-based and policy-based RL methods. Our benchmark consists of 8 different language tasks, which require multiple rounds of language interaction and cover a range of tasks in open-ended dialogue and text games.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 57 of its 91 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

57 of 91 references.

References · 57

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2020Learning to summarize from human feedbackcited by 2,562
38 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  8. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  9. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  10. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  11. 2023Tool Learning with Foundation Modelscited by 302
  12. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  13. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  14. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  15. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  16. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  17. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  18. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  19. 2019Interactive Fiction Games: A Colossal Adventurecited by 165
  20. 2018Decoupling Strategy and Generation in Negotiation Dialoguescited by 156
  21. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  22. 2020Keep CALM and Explore: Language Models for Action Generation in Text-based Gamescited by 108
  23. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  24. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  25. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  26. 2019Better Rewards Yield Better Summaries: Learning to Summarise Without Referencescited by 76
  27. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 71
  28. 2015Language Understanding for Text-based Games Using Deep Reinforcement Learningcited by 70
  29. 2020Text Generation by Learning from Demonstrationscited by 64
  30. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 63
  31. 2023Rewarding Chatbots for Real-World Engagement with Millions of Userscited by 58
  32. 2021What Would Jiminy Cricket Do? Towards Agents That Behave Morallycited by 57
  33. 2018Controllable Neural Story Plot Generation via Reward Shapingcited by 49
  34. 2020How to Avoid Being Eaten by a Grue: Structured Exploration Strategies for Textual Worldscited by 38
  35. 2022CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learningcited by 33
  36. 2021Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 32
  37. 2020Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learningcited by 27
  38. 2017Trainable Greedy Decoding for Neural Machine Translationcited by 23

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 17; 2025: 38; 2026: 28.

84 papers.

Sort:

By area:

  1. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  2. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  3. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  4. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  5. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  6. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 117
  7. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 94
  8. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agentscited by 89
  9. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
  10. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 80
  11. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
  12. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
38 more
  1. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  2. 2025STeCa: Step-level Trajectory Calibration for LLM Agent Learningcited by 37
  3. 2023LLF-Bench: Benchmark for Interactive Learning from Language Feedbackcited by 30
  4. 2024DynaSaur: Large Language Agents Beyond Predefined Actionscited by 30
  5. 2024How Far Are We from Intelligent Visual Deductive Reasoning?cited by 29
  6. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 29
  7. 2025Training a Generally Curious Agentcited by 29
  8. 2025ATLaS: Agent Tuning via Learning Critical Stepscited by 29
  9. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?cited by 24
  10. 2025TextArenacited by 24
  11. 2025From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?cited by 21
  12. 2025Fine-Grained GRPO for Precise Preference Alignment in Flow Modelscited by 21
  13. 2025Digi-Q: Learning Q-Value Functions for Training Device-Control Agentscited by 20
  14. 2024Natural Language Reinforcement Learningcited by 17
  15. 2025GEM: A Gym for Agentic LLMscited by 17
  16. 2025Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Modelscited by 16
  17. 2024Q-Probe: A Lightweight Approach to Reward Maximization for Language Modelscited by 15
  18. 2025ETS: Efficient Tree Search for Inference-Time Scalingcited by 15
  19. 2024Efficacy of Language Model Self-Play in Non-Zero-Sum Gamescited by 14
  20. 2025GRE Suite: Geo-localization Inference via Fine-Tuned Vision-Language Models and Enhanced Reasoning Chainscited by 14
  21. 2024Dialogue Action Tokens: Steering Language Models in Goal-Directed Dialogue with a Multi-Turn Plannercited by 12
  22. 2026Confidence Estimation for LLMs in Multi-turn Interactionscited by 12
  23. 2025A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoningcited by 11
  24. 2025PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learningcited by 11
  25. 2026ARLArena: A Unified Framework for Stable Agentic Reinforcement Learningcited by 11
  26. 2025ARIA: Training Language Agents with Intention-Driven Reward Aggregationcited by 9
  27. 2025General Modular Harness for LLM Agents in Multi-Turn Gaming Environmentscited by 9
  28. 2025Multi-Turn Puzzles: Evaluating Interactive Reasoning and Strategic Dialogue in LLMscited by 9
  29. 2026VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agentscited by 9
  30. 2025InfoQuest: Evaluating Multi-Turn Dialogue Agents for Open-Ended Conversations with Hidden Contextcited by 8
  31. 2025COMPASS: Benchmarking Constrained Optimization in LLM Agentscited by 8
  32. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  33. 2025TextAtari: 100K Frames Game Playing with Language Agentscited by 6
  34. 2025Collaborate, Deliberate, Evaluate: How LLM Alignment Affects Coordinated Multi-Agent Outcomescited by 6
  35. 2026PABU: Progress-Aware Belief Update for Efficient LLM Agentscited by 6
  36. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspectivecited by 5
  37. 2025PuzzlePlex: Benchmarking Foundation Models on Reasoning and Planning with Puzzlescited by 5
  38. 2026Mitigating Conversational Inertia in Multi-Turn Agentscited by 5

50 most cited of 84 All 84 (JSON)