mlsanityData to 7 Oct 2026

Language Alignment via Nash-learning and Adaptive feedback

arXiv 2406.15890 ↗First posted 22 Jun 2024cs.LG cs.AI cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Recent research has shown the potential of Nash Learning via Human Feedback for large language model alignment by incorporating the notion of a preference model in a minimax game setup. We take this idea further by casting the alignment as a mirror descent algorithm against the adaptive feedback of an improved opponent, thereby removing the need for learning a preference model or the existence of an annotated dataset altogether. The resulting algorithm, which we refer to as Language Alignment via Nash-learning and Adaptive feedback (LANA), is capable of self-alignment without the need for a human-annotated preference dataset. We support this statement with various experiments and mathematical discussion.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 20 references that are in the map. First posted on arXiv 22 Jun 2024.

Builds on

13 of 20 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  2. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  3. 2024Self-Rewarding Language Models18 Jan
  4. 2023Nash Learning from Human Feedback1 Dec
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 7

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023Mistral 7Bcited by 3,065
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  6. 2023Textbooks Are All You Needcited by 561
  7. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 14
  2. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching
  2. 2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences
  3. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descent
  4. 2024Human Alignment of Large Language Models through Online Preference Optimisation
  5. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning
  6. 2025Unregularized Linear Convergence in Zero-Sum Game from Preference Feedback
  7. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  8. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment