mlsanityData to 14 Oct 2026

Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment

arXiv 2502.18099 ↗First posted 25 Feb 2025cs.LGcited by 10

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models (LLMs) with human preferences typically demands vast amounts of meticulously curated data, which is both expensive and prone to labeling noise. We propose Stackelberg Game Preference Optimization (SGPO), a robust alignment framework that models alignment as a two-player Stackelberg game between a policy (leader) and a worst-case preference distribution (follower). The proposed SGPO guarantees O(ε)-bounded regret within an ε-Wasserstein ball, offering formal robustness to (self-)annotation noise. We instantiate SGPO with Stackelberg Self-Annotated Preference Optimization (SSAPO), which uses minimal human-labeled "seed" preferences and iteratively self-annotates new prompts. In each iteration, SSAPO applies a distributionally robust reweighting of synthetic annotations, ensuring that noisy or biased self-labels do not derail training. Remarkably, using only 2K seed preferences -- about 1/30 of standard human labels -- SSAPO achieves strong win rates against GPT-4 across multiple benchmarks within three iterations. These results highlight that a principled Stackelberg formulation yields data-efficient alignment for LLMs, significantly reducing reliance on costly human annotations.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 47 references that are in the map. First posted on arXiv 25 Feb 2025.

Builds on

32 of 47 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Distributional Preference Alignment of LLMs via Optimal Transport9 Jun
  2. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment6 Jun
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  4. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  5. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 25

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023Mistral 7Bcited by 3,164
  7. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  10. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  11. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  12. 2024Self-Rewarding Language Modelscited by 681
13 more
  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  2. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  3. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  4. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  5. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  6. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  7. 2015Data-driven Distributionally Robust Optimization Using the Wasserstein Metric: Performance Guarantees and Tractable Reformulationscited by 291
  8. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  9. 2023Nash Learning from Human Feedbackcited by 224
  10. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  11. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  12. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  13. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 9.

10 papers.

  1. 2025Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Gamecited by 10
  2. 2026FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learningcited by 8
  3. 2026Explaining and Preventing Alignment Collapse in Iterative RLHFcited by 4
  4. 2026Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspectivecited by 2
  5. 2026AnomSeer: Reinforcing Multimodal LLMs to Reason for Time-Series Anomaly Detectioncited by 0
  6. 2026A Markov Chain Approach to Preference Alignmentcited by 0
  7. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0
  8. 2026Multi-LLM Collaborative Alignment via Stackelberg Gamescited by 0
  9. 2026Robust Nash Alignment under Preference Uncertaintycited by 0
  10. 2026Can You Trust Your Preference Models? Stackelberg Games for Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
  2. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  3. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
  4. 2026Reliability-Aware LLM Alignment from Inconsistent Human Feedback
  5. 2024Sample-Efficient Alignment for LLMs
  6. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  7. 2025RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment
  8. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models