mlsanityData to 14 Oct 2026

Offline RL for Natural Language Generation with Implicit Language Q Learning

arXiv 2206.11871 ↗First posted 5 Jun 2022cs.CL cs.LGcited by 138

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

Large language models distill broad knowledge from text corpora. However, they can be inconsistent when it comes to completing user specified tasks. This issue can be addressed by finetuning such models via supervised learning on curated datasets, or via reinforcement learning. In this work, we propose a novel offline RL method, implicit language Q-learning (ILQL), designed for use on language models, that combines both the flexible utility maximization framework of RL algorithms with the ability of supervised learning to leverage previously collected data, as well as its simplicity and stability. Our method employs a combination of value conservatism alongside an implicit dataset support constraint in learning value functions, which are then used to guide language model generations towards maximizing user-specified utility functions. In addition to empirically validating ILQL, we present a detailed empirical analysis of situations where offline RL can be useful in natural language generation settings, demonstrating how it can be a more effective utility optimizer than prior approaches for end-to-end dialogue, and how it can effectively optimize high variance reward functions based on subjective judgement, such as whether to label a comment as toxic or not.

Key facts. Cited by 138 papers in the map (data through 14 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 60 of its 74 references that are in the map. First posted on arXiv 5 Jun 2022.

Builds on

60 of 74 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning18 Apr
  2. 2020Human-centric Dialog Training via Offline Reinforcement Learning12 Oct
  3. 2020Text Generation by Learning from Demonstrations16 Sep

Other references · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2020Learning to summarize from human feedbackcited by 2,588
  11. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
38 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  6. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  9. 2018Hierarchical Neural Story Generationcited by 1,193
  10. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  12. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  13. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  14. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  15. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  16. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  17. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  18. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  19. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 376
  20. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  21. 2020Critic Regularized Regressioncited by 275
  22. 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 263
  23. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  24. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 165
  25. 2021Offline RL Without Off-Policy Evaluationcited by 151
  26. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  27. 2018Learning to Write with Cooperative Discriminatorscited by 119
  28. 2020If beam search is the answer, what was the question?cited by 116
  29. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 115
  30. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 101
  31. 2016Learning to Translate in Real-time with Neural Machine Translationcited by 97
  32. 2019Better Rewards Yield Better Summaries: Learning to Summarise Without Referencescited by 77
  33. 2017Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learningcited by 71
  34. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 64
  35. 2021Machine Translation Decoding beyond Beam Searchcited by 55
  36. 2018Controllable Neural Story Plot Generation via Reward Shapingcited by 49
  37. 2018Generating More Interesting Responses in Neural Conversation Models with Distributional Constraintscited by 33
  38. 2016On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systemscited by 28

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 42; 2024: 43; 2025: 36; 2026: 15.

138 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  3. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
  4. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  5. 2023Augmented Language Models: a Surveycited by 403
  6. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  7. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 336
  8. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  9. 2023Pretraining Language Models with Human Preferencescited by 247
  10. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  11. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  12. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
38 more
  1. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  2. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 165
  3. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  4. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  5. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  6. 2023Controlled Decoding from Language Modelscited by 135
  7. 2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimizationcited by 112
  8. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 111
  9. 2024ARGS: Alignment as Reward-Guided Searchcited by 103
  10. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 90
  11. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 88
  12. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 67
  13. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
  14. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  15. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 56
  16. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  17. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  18. 2023Learning to Generate Better Than Your LLMcited by 49
  19. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  20. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignmentcited by 48
  21. 2024Transfer Q Star: Principled Decoding for LLM Alignmentcited by 48
  22. 2022Critic-Guided Decoding for Controlled Text Generationcited by 45
  23. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  24. 2024Moral Alignment for LLM Agentscited by 39
  25. 2023Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversationscited by 38
  26. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
  27. 2023Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuningcited by 33
  28. 2023Interactive Natural Language Processingcited by 31
  29. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  30. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  31. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 30
  32. 2024Efficient Reinforcement Learning with Large Language Model Priorscited by 30
  33. 2025Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimizationcited by 29
  34. 2025SWE-Dev: Building Software Engineering Agents with Training and Inference Scalingcited by 26
  35. 2023SPRINT: Scalable Policy Pre-Training via Language Instruction Relabelingcited by 24
  36. 2024Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimizationcited by 22
  37. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Modelscited by 21
  38. 2023Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systemscited by 19

50 most cited of 138 All 138 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving Neutral Point-of-View Generation with Data- and Parameter-Efficient RL
  2. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  4. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
  5. 2026Reinforcement Learning from Meta-Evaluation: Aligning Language Models Without Ground-Truth Labels
  6. 2025Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study
  7. 2025Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels
  8. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback