mlsanityData to 7 Oct 2026

Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models

arXiv 2305.14718 ↗First posted 24 May 2023cs.CLcited by 17

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning with Human Feedback (RLHF) is the most prominent method for Language Model (LM) alignment. However, RLHF is an unstable and data-hungry process that continually requires new high-quality LM-generated data for finetuning. We introduce Advantage-Leftover Lunch RL (A-LoL), a new class of offline policy gradient algorithms that enable RL training on any pre-existing data. By assuming the entire LM output sequence as a single action, A-LoL allows incorporating sequence-level classifiers or human-designed scoring functions as rewards. Subsequently, by using LM's value estimate, A-LoL only trains on positive advantage (leftover) data points, making it resilient to noise. Overall, A-LoL is an easy-to-implement, sample-efficient, and stable LM training recipe. We demonstrate the effectiveness of A-LoL and its variants with a set of four different language generation tasks. We compare against both online RL (PPO) and recent preference-based (DPO, PRO) and reward-based (GOLD) offline RL baselines. On the commonly-used RLHF benchmark, Helpful and Harmless Assistant (HHA), LMs trained with A-LoL methods achieve the highest diversity while also being rated more safe and helpful than the baselines according to humans. Additionally, in the remaining three tasks, A-LoL could optimize multiple distinct reward functions even when using noisy or suboptimal training data. We also release our experimental code. https://github.com/abaheti95/LoL-RL

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 66 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

37 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Aligning Language Models with Preferences through f-divergence Minimization16 Feb
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  8. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  11. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  12. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
23 more
  1. 2019Measurement and Fairnesscited by 299
  2. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  3. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  4. 2020Critic Regularized Regressioncited by 269
  5. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  6. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  7. 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
  8. 2022On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?cited by 137
  9. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  10. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
  11. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  12. 2022Measuring the Carbon Intensity of AI in Cloud Instancescited by 110
  13. 2022FaithDial: A Faithful Benchmark for Information-Seeking Dialoguecited by 108
  14. 2016Reward Augmented Maximum Likelihood for Neural Structured Predictioncited by 80
  15. 2020Dialogue Response Ranking Training with Large-Scale Human Feedback Datacited by 68
  16. 2020Text Generation by Learning from Demonstrationscited by 64
  17. 2018Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Taskcited by 56
  18. 2021Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contextscited by 56
  19. 2018Toward Diverse Text Generation with Inverse Reinforcement Learningcited by 41
  20. 2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 35
  21. 2022CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learningcited by 33
  22. 2023Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systemscited by 19
  23. 2021Causal-aware Safe Policy Improvement for Task-oriented dialoguecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 3; 2025: 6; 2026: 4.

17 papers.

Sort:

By area:

  1. 2023Large Language Model Alignment: A Surveycited by 241
  2. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 80
  3. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 67
  4. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  5. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
  6. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  7. 2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignmentcited by 14
  8. 2023Reinforcement Learning for Generative AI: A Surveycited by 11
  9. 2023Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challengescited by 10
  10. 2025Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimizationcited by 10
  11. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 10
  12. 2024Varying Shades of Wrong: Aligning LLMs with Wrong Answers Onlycited by 9
5 more
  1. 2025One Framework to Rule Them All: Unifying RL-Based and RL-Free Methods in RLHFcited by 2
  2. 2025Bootstrapping LLMs via Preference-Based Policy Optimizationcited by 0
  3. 2026Robust Post-Training for Generative Recommenders: Why Exponential Reward-Weighted SFT Outperforms RLHFcited by 0
  4. 2026Block-R1: Rethinking the Role of Block Size in Multi-domain Reinforcement Learning for Diffusion Large Language Modelscited by 0
  5. 2026EmoDistill: Offline Emotion Skill Distillation for Language Model Agents in Adversarial Negotiationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignment
  2. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  3. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
  4. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards
  5. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  6. 2023Aligning Language Models with Offline Learning from Human Feedback
  7. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  8. 2024Understanding the performance gap between online and offline alignment algorithms