mlsanityData to 14 Oct 2026

Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains

arXiv 2501.05707 ↗First posted 10 Jan 2025cs.CL cs.AI cs.LGcited by 95

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Large language models (LLMs) have achieved remarkable performance in recent years but are fundamentally limited by the underlying training data. To improve models beyond the training data, recent works have explored how LLMs can be used to generate synthetic data for autonomous self-improvement. However, successive steps of self-improvement can reach a point of diminishing returns. In this work, we propose a complementary approach towards self-improvement where finetuning is applied to a multiagent society of language models. A group of language models, all starting from the same base model, are independently specialized by updating each one using data generated through multiagent interactions among the models. By training each model on independent sets of data, we illustrate how this approach enables specialization across models and diversification over the set of models. As a result, our overall system is able to preserve diverse reasoning chains and autonomously improve over many more rounds of fine-tuning than single-agent self-improvement methods. We quantitatively illustrate the efficacy of the approach across a wide suite of reasoning tasks.

Key facts. Cited by 95 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 41 of its 48 references that are in the map. First posted on arXiv 10 Jan 2025.

Builds on

41 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Let Models Speak Ciphers: Multiagent Debate through Embeddings10 Oct
  2. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate30 May

Other references · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2017Deep reinforcement learning from human preferencescited by 3,912
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  12. 2023Mistral 7Bcited by 3,164
27 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  2. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  3. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  4. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  5. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  6. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  7. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  9. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  10. 2023Instruction Tuning with GPT-4cited by 686
  11. 2024Self-Rewarding Language Modelscited by 681
  12. 2022Large Language Models Can Self-Improvecited by 664
  13. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  14. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  15. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  16. 2022Generating Sequences by Learning to Self-Correctcited by 277
  17. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  18. 2024Iterative Reasoning Preference Optimizationcited by 246
  19. 2023Large Language Model Guided Tree-of-Thoughtcited by 221
  20. 2022Formal Mathematics Statement Curriculum Learningcited by 142
  21. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoningcited by 86
  22. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 70
  23. 2024Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraintcited by 39
  24. 2023The Consensus Game: Language Model Generation via Equilibrium Searchcited by 36
  25. 2023Teaching Language Models to Self-Improve through Interactive Demonstrationscited by 26
  26. 2021Iterated learning for emergent systematicity in VQAcited by 21
  27. 2023Regularized Conventions: Equilibrium Computation as a Model of Pragmatic Reasoningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 59; 2026: 36.

95 papers.

Sort:

By area:

  1. 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automationcited by 131
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  3. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  4. 2025Agentic Large Language Models, a surveycited by 108
  5. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 79
  6. 2025Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debatecited by 66
  7. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 62
  8. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 52
  9. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  10. 2025Inference-time Scaling of Diffusion Models through Classical Searchcited by 49
  11. 2025LLM Collaboration With Multi-Agent Reinforcement Learningcited by 49
  12. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 46
38 more
  1. 2025SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoningcited by 44
  2. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 39
  3. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 38
  4. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 36
  5. 2025Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routingcited by 34
  6. 2025OSC: Cognitive Orchestration through Dynamic Knowledge Alignment in Multi-Agent LLM Collaborationcited by 30
  7. 2025Thought Communication in Multiagent Collaborationcited by 30
  8. 2025Free-MAD: Consensus-Free Multi-Agent Debatecited by 29
  9. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 28
  10. 2025Large Language Models Miss the Multi-Agent Markcited by 26
  11. 2025Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Informationcited by 26
  12. 2025Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systemscited by 23
  13. 2025Emergent Coordination in Multi-Agent Language Modelscited by 23
  14. 2025MaskSearch: A Universal Pre-Training Framework to Enhance Agentic Search Capabilitycited by 22
  15. 2026LatentMem: Customizing Latent Memory for Multi-Agent Systemscited by 15
  16. 2025Don't lie to your friends: Learning what you know from collaborative self-playcited by 14
  17. 2025MASLab: A Unified and Comprehensive Codebase for LLM-based Multi-Agent Systemscited by 14
  18. 2025Controlling Performance and Budget of a Centralized Multi-agent LLM System with Reinforcement Learningcited by 13
  19. 2025Seven Security Challenges in Cross-domain Multi-agent LLM Systemscited by 12
  20. 2025Temporal Consistency for LLM Reasoning Process Error Identificationcited by 10
  21. 2025MAD-Spear: A Conformity-Driven Prompt Injection Attack on Multi-Agent Debate Systemscited by 10
  22. 2026LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agentscited by 10
  23. 2026Recursive Multi-Agent Systemscited by 10
  24. 2025Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMscited by 9
  25. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 9
  26. 2025Stochastic Self-Organization in Multi-Agent Systemscited by 9
  27. 2025Two Heads Are Better Than One: Dual-Model Verbal Reflection at Inference-Timecited by 8
  28. 2026Learning Decentralized LLM Collaboration with Multi-Agent Actor Criticcited by 8
  29. 2026ICaRus: Identical Cache Reuse for Efficient Multi Model Inferencecited by 8
  30. 2025Self-Improvement of Language Models by Post-Training on Multi-Agent Debatecited by 7
  31. 2026PrivAct: Internalizing Contextual Privacy Preservation via Multi-Agent Preference Trainingcited by 7
  32. 2026MoCo: A One-Stop Shop for Model Collaboration Researchcited by 6
  33. 2025Interactive Learning for LLM Reasoningcited by 5
  34. 2025GuidedSampling: Steering LLMs Towards Diverse Candidate Solutions at Inference-Timecited by 5
  35. 2025AlphaApollo: A System for Deep Agentic Reasoningcited by 5
  36. 2025Learning Steerable Clarification Policies with Collaborative Self-playcited by 5
  37. 2026Latent Agents: A Post-Training Procedure for Internalized Multi-Agent Debatecited by 5
  38. 2026Economy of Minds: Emerging Multi-Agent Intelligence with Economic Interactionscited by 5

50 most cited of 95 All 95 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
  2. 2025Self-Improving LLM Agents at Test-Time
  3. 2026From Solo to Social Learning: Characterizing Recursive Social Improvement in LLMs
  4. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
  5. 2024Can Large Language Models Invent Algorithms to Improve Themselves?: Algorithm Discovery for Recursive Self-Improvement through Reinforcement Learning
  6. 2025Multi-Agent Evolve: LLM Self-Improve through Co-evolution
  7. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
  8. 2024MALT: Improving Reasoning with Multi-Agent LLM Training