mlsanityData to 7 Oct 2026

Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models

arXiv 2406.04274 ↗First posted 6 Jun 2024cs.LG cs.AI cs.CLcited by 16

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 2 more
Abstract (arXiv)

This work studies the challenge of aligning large language models (LLMs) with offline preference data. We focus on alignment by Reinforcement Learning from Human Feedback (RLHF) in particular. While popular preference optimization methods exhibit good empirical performance in practice, they are not theoretically guaranteed to converge to the optimal policy and can provably fail when the data coverage is sparse by classical offline reinforcement learning (RL) results. On the other hand, a recent line of work has focused on theoretically motivated preference optimization methods with provable guarantees, but these are not computationally efficient for large-scale applications like LLM alignment. To bridge this gap, we propose SPAC, a new offline preference optimization method with self-play, inspired by the on-average pessimism technique from the offline RL literature, to be the first provable and scalable approach to LLM alignment. We both provide theoretical analysis for its convergence under single-policy concentrability for the general function approximation setting and demonstrate its competitive empirical performance for LLM alignment on a 7B Mistral model with Open LLM Leaderboard evaluations.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 57 of its 85 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

57 of 85 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  2. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 54

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
38 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  5. 2019Introduction to Online Convex Optimizationcited by 793
  6. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  7. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  8. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  9. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  10. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  11. 2019A Theory of Regularized Markov Decision Processescited by 279
  12. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  13. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  14. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  15. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  16. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  17. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  18. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  19. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  20. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  21. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  22. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  23. 2017Emergent Complexity via Multi-Agent Competitioncited by 117
  24. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  25. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  26. 2020Batch Value-function Approximation with Only Realizabilitycited by 103
  27. 2015Contextual Dueling Banditscited by 101
  28. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  29. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learningcited by 82
  30. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  31. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  32. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  33. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  34. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 63
  35. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  36. 2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
  37. 2022The Efficacy of Pessimism in Asynchronous Q-Learningcited by 41
  38. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 7.

16 papers.

Sort:

By area:

  1. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  2. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  3. 2025Pluralistic Off-policy Evaluation and Alignmentcited by 11
  4. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Explorationcited by 8
  5. 2025Learning a Pessimistic Reward Model in RLHFcited by 5
  6. 2025KnowRL: Teaching Language Models to Know What They Knowcited by 5
  7. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferencescited by 4
  8. 2026Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspectivecited by 2
  9. 2025Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Datacited by 1
  10. 2025Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignmentcited by 1
  11. 2026Breaking the Capability Ceiling of LLM Post-Training by Reintroducing Markov Statescited by 1
  12. 2026Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distributioncited by 0
4 more
  1. 2026IRIS: Interpolative R'enyi Iterative Self-play for Large Language Model Fine-Tuningcited by 0
  2. 2026S-SPPO: Semantic-Calibrated Self-Play Preference Optimizationcited by 0
  3. 2026When Does Trajectory-Level Supervision Permit Efficient Offline Reinforcement Learning?cited by 0
  4. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  2. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  3. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models
  4. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment
  5. 2024Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment
  6. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descent
  7. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
  8. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models