mlsanityData to 7 Oct 2026

Self-Generative Adversarial Fine-Tuning for Large Language Models

arXiv 2602.01137 ↗First posted 1 Feb 2026cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Fine-tuning large language models (LLMs) for alignment typically relies on supervised fine-tuning or reinforcement learning from human feedback, both limited by the cost and scarcity of high-quality annotations. Recent self-play and synthetic data approaches reduce this dependence but often rely on heuristic assumptions or ungrounded self-evaluation, which can cause bias accumulation and performance drift. In this paper, we propose Self-Generative Adversarial LLM (SGALM), a unified fine-tuning framework that formulates alignment as a generative adversarial game within a single LLM. SGALM jointly evolves generation and discrimination capabilities without external reward models. Theoretical and empirical results demonstrate that SGALM achieves state-of-the-art performance, serves as an effective alignment algorithm and a robust synthetic data engine.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 41 references that are in the map. First posted on arXiv 1 Feb 2026.

Builds on

32 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Aligning Language Models with Preferences through f-divergence Minimization16 Feb
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  10. 2022Red Teaming Language Models with Language Modelscited by 983
  11. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  12. 2018Which Training Methods for GANs do actually Converge?cited by 672
17 more
  1. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  2. 2024Self-Rewarding Language Modelscited by 654
  3. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  4. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  5. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  6. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  8. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  9. 2017Data Augmentation Generative Adversarial Networkscited by 278
  10. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  11. 2024Iterative Reasoning Preference Optimizationcited by 244
  12. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  13. 2017Long Text Generation via Adversarial Training with Leaked Informationcited by 100
  14. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  15. 2017Adversarial Feature Matching for Text Generationcited by 58
  16. 2017Maximum-Likelihood Augmented Discrete Generative Adversarial Networkscited by 39
  17. 2023Fine-tuning Language Models with Generative Adversarial Reward Modellingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026IRIS: Interpolative R'enyi Iterative Self-play for Large Language Model Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  2. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  3. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game
  4. 2023Large Language Model Alignment: A Survey
  5. 2026LLMs as High-Dimensional Nonlinear Autoregressive Models with Attention: Training, Alignment and Inference
  6. 2025Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning
  7. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning
  8. 2026Adversarial Alignment: Ensuring Value Consistency in Large Language Models for Sensitive Domains