mlsanityData to 7 Oct 2026

Latent Adversarial Regularization for Offline Preference Optimization

arXiv 2601.22083 ↗First posted 29 Jan 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Learning from human feedback typically relies on preference optimization that constrains policy updates through token-level regularization. However, preference optimization for language models is particularly challenging because token-space similarity does not imply semantic or behavioral similarity. To address this challenge, we leverage latent-space regularization for language model preference optimization. We introduce GANPO, which achieves latent-space regularization by penalizing divergence between the internal representations of a policy model and a reference model. Given that latent representations are not associated with explicit probability densities, we adopt an adversarial approach inspired by GANs to minimize latent-space divergence. We integrate GANPO as a regularizer into existing offline preference optimization objectives. Experiments across multiple model architectures and tasks show consistent improvements from latent-space regularization. Further, by comparing GANPO-induced inferential biases with those from token-level regularization, we find that GANPO provides more robust structural feedback under distributional shift and noise while maintaining comparable downstream performance with minor computational overhead.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 56 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

46 of 56 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Length Desensitization in Direct Preference Optimization10 Sep
  2. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization18 Jul
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  4. 2024sDPO: Don't Use Your Data All at Once28 Mar
  5. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game14 Nov
  6. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  7. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  8. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2017Improved Training of Wasserstein GANscited by 2,223
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  9. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
25 more
  1. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2024Self-Rewarding Language Modelscited by 654
  4. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  5. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  6. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  7. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  8. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  9. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  10. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  11. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  12. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  13. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  14. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  15. 2018The relativistic discriminator: a key element missing from standard GANcited by 234
  16. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  17. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  18. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  19. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  20. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 101
  21. 2025Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thoughtcited by 73
  22. 2017Adversarial Feature Matching for Text Generationcited by 58
  23. 2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
  24. 2020TextGAIL: Generative Adversarial Imitation Learning for Text Generationcited by 28
  25. 2019On Relativistic f-Divergencescited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback
  2. 2025WorldPM: Scaling Human Preference Modeling
  3. 2026Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment
  4. 2025Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
  5. 2025AAPA: Adversarially Anchored Preference Alignment for Post-Training of Large Language Models
  6. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
  7. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  8. 2025GAPO: Learning Preferential Prompt through Generative Adversarial Policy Optimization