mlsanityData to 14 Oct 2026

Sample-Efficient Alignment for LLMs

arXiv 2411.01493 ↗First posted 3 Nov 2024cs.LG cs.AI cs.CLcited by 16

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We study methods for efficiently aligning large language models (LLMs) with human preferences given budgeted online feedback. We first formulate the LLM alignment problem in the frame of contextual dueling bandits. This formulation, subsuming recent paradigms such as online RLHF and online DPO, inherently quests for sample-efficient algorithms that incorporate online active exploration. Leveraging insights from bandit theory, we introduce a unified algorithm based on Thompson sampling and highlight its applications in two distinct LLM alignment scenarios. The practical agent that efficiently implements this algorithm, named SEA (Sample-Efficient Alignment), is empirically validated through extensive experiments across three model scales (1B, 2.8B, 6.9B) and three preference learning algorithms (DPO, IPO, SLiC). The results demonstrate that SEA achieves highly sample-efficient alignment with oracle's preferences, outperforming recent active exploration methods for LLMs. Additionally, we release the implementation of SEA together with an efficient codebase designed for online alignment of LLMs, aiming to accelerate future research in this field.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 59 of its 86 references that are in the map. First posted on arXiv 3 Nov 2024.

Builds on

59 of 86 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Bootstrapping Language Models with DPO Implicit Rewards14 Jun
  2. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment29 May
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  4. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  5. 2024Active Preference Learning for Large Language Models12 Feb
  6. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  7. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  12. 2020Learning to summarize from human feedbackcited by 2,588
38 more
  1. 2015Trust Region Policy Optimizationcited by 2,580
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  9. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  10. 2017A Tutorial on Thompson Samplingcited by 433
  11. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  13. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  14. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  15. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  16. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  17. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  18. 2016Simple Bayesian Algorithms for Best Arm Identificationcited by 117
  19. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  20. 2015Contextual Dueling Banditscited by 101
  21. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 96
  22. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  23. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  24. 2017Preferential Bayesian Optimizationcited by 74
  25. 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 62
  26. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarizationcited by 50
  27. 2020An Empirical Process Approach to the Union Bound: Practical Algorithms for Combinatorial and Linear Banditscited by 45
  28. 2024Efficient Exploration for LLMscited by 45
  29. 2016Double Thompson Sampling for Dueling Banditscited by 42
  30. 2020Hypermodels for Explorationcited by 37
  31. 2022Contextual Bandits with Large Action Spaces: Made Practicalcited by 33
  32. 2024Feel-Good Thompson Sampling for Contextual Dueling Banditscited by 27
  33. 2022An Analysis of Ensemble Samplingcited by 24
  34. 2024Deep Bayesian Active Learning for Preference Modeling in Large Language Modelscited by 19
  35. 2022The Alberta Plan for AI Researchcited by 18
  36. 2021Launchpad: A Programming Model for Distributed Machine Learning Researchcited by 14
  37. 2024Locality Sensitive Sparse Encoding for Learning World Models Onlinecited by 12
  38. 2022Efficient Offline Policy Optimization with a Learned Modelcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 6.

16 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025PILAF: Optimal Human Preference Sampling for Reward Modelingcited by 20
  3. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 18
  4. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimizationcited by 17
  5. 2025Efficient Process Reward Model Training via Active Learningcited by 16
  6. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  7. 2025ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignmentcited by 10
  8. 2025SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learningcited by 6
  9. 2025Inference-Time Personalized Alignment with a Few User Preference Queriescited by 6
  10. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Modelscited by 3
  11. 2025General Exploratory Bonus for Optimistic Exploration in RLHFcited by 2
  12. 2026LAD: Learning Advantage Distribution for Reasoningcited by 2
4 more
  1. 2026ActiveUltraFeedback: Efficient Preference Data Generation using Active Learningcited by 2
  2. 2026A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Banditscited by 1
  3. 2026Active Learning for Stochastic Contextual Linear Banditscited by 1
  4. 2026Efficient Exploration at Scalecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models
  2. 2023Sample Efficient Preference Alignment in LLMs via Active Exploration
  3. 2025Online Preference Alignment for Language Models via Count-based Exploration
  4. 2025ComPO: Preference Alignment via Comparison Oracles
  5. 2026A Zeroth-Order Paradigm for LLM Preference Alignment
  6. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  7. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  8. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization