mlsanityData to 7 Oct 2026

Multi-LLM Collaborative Alignment via Stackelberg Games

arXiv 2609.39076 ↗First posted 30 Sep 2026cs.AI cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods also fits: Process reward models for reasoning

Abstract (arXiv)

A pool of language models can collaborate and improve collectively by learning from one another's responses. These interactions depend on the instructions used during training. Existing methods typically sample instructions uniformly, even though their usefulness may change as the models improve: an instruction on which models' responses once differed in quality may later be answered equally well, while a previously difficult instruction may begin to provide a useful learning signal. We propose Stackelberg Alignment, a game-theory-inspired leader-follower framework that turns instruction selection into an adaptive curriculum. An EXP3 bandit acts as the leader, allocating a fixed sampling budget across instructions and updating its sampling distribution using a reward that combines instruction difficulty and response discriminability. The language models act as followers: they respond to the selected instructions, evaluate one another's responses, and learn from the resulting preference signals through DPO or GRPO. The framework uses Elo-style reputation-weighted peer judgment and reputation-based opponent matching to support reliable and competitive model interactions. Experiments across three heterogeneous model pools and 12 benchmarks spanning scientific discovery, reasoning, code, instruction following, and knowledge show that Stackelberg Alignment achieves the highest macro-average across three diverse model pools, outperforming the strongest training-time baseline by up to 7.4% and the best static inference baseline by 12-25%. Analysis confirms that the adaptive leader concentrates duels on the most informative instructions, and ablations show that both reputation-weighted judgment and reputation-based matching improve the effectiveness of multi-LLM evolution.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 54 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

46 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combat5 Jun
  2. 2025Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment25 Feb

Other references · 44

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  10. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  11. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  12. 2024Phi-4 Technical Reportcited by 675
32 more
  1. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  3. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  4. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  5. 2023Nash Learning from Human Feedbackcited by 220
  6. 2026Gemma 4 Technical Reportcited by 187
  7. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 154
  8. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 138
  9. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 121
  10. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  11. 2025Self-Evolving Curriculum for LLM Reasoningcited by 107
  12. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  13. 2025Multiagent Finetuning: Self Improvement with Diverse Reasoning Chainscited by 90
  14. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
  15. 2025BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biologycited by 75
  16. 2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Usecited by 67
  17. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 63
  18. 2025NVIDIA Nemotron 3: Efficient and Open Intelligencecited by 54
  19. 2025The Majority is not always right: RL training for solution aggregationcited by 46
  20. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  21. 2024CREAM: Consistency Regularized Self-Rewarding Language Modelscited by 35
  22. 2024CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeamingcited by 27
  23. 2024Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arenacited by 26
  24. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 25
  25. 2025Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systemscited by 22
  26. 2025CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMscited by 16
  27. 2024Contextual Bilevel Reinforcement Learning for Incentive Alignmentcited by 14
  28. 2026Among Us: Measuring and Mitigating Malicious Contributions in Model Collaboration Systemscited by 6
  29. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 6
  30. 2026SMDD-Bench: Can LLMs Solve Real-World Small Molecule Drug Design Tasks?cited by 4
  31. 2026AssayBench: An Assay-Level Virtual Cell Benchmark for LLMs and Agentscited by 2
  32. 2026Scaling Participation in Modular AI Systemscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Elo-Evolve: A Co-evolutionary Framework for Language Model Alignment
  2. 2024Toward Optimal LLM Alignments Using Two-Player Games
  3. 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction following
  4. 2025GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfare
  5. 2024Sample-Efficient Alignment for LLMs
  6. 2024Decoupled Alignment for Robust Plug-and-Play Adaptation
  7. 2024CoEvol: Constructing Better Responses for Instruction Finetuning through Multi-Agent Cooperation
  8. 2026SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utility