mlsanityData to 14 Oct 2026

Synergistic Weak-Strong Collaboration by Aligning Preferences

arXiv 2504.15188 ↗First posted 21 Apr 2025cs.AIcited by 2

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Current Large Language Models (LLMs) excel in general reasoning yet struggle with specialized tasks requiring proprietary or domain-specific knowledge. Fine-tuning large models for every niche application is often infeasible due to black-box constraints and high computational overhead. To address this, we propose a collaborative framework that pairs a specialized weak model with a general strong model. The weak model, tailored to specific domains, produces initial drafts and background information, while the strong model leverages its advanced reasoning to refine these drafts, extending LLMs' capabilities to critical yet specialized tasks. To optimize this collaboration, we introduce a collaborative feedback to fine-tunes the weak model, which quantifies the influence of the weak model's contributions in the collaboration procedure and establishes preference pairs to guide preference tuning of the weak model. We validate our framework through experiments on three domains. We find that the collaboration significantly outperforms each model alone by leveraging complementary strengths. Moreover, aligning the weak model with the collaborative preference further enhances overall performance.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 29 of its 41 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

29 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  12. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
16 more
  1. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  3. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  4. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  5. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  6. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 251
  7. 2023Large Language Model Routing with Benchmark Datasetscited by 172
  8. 2022ProsocialDialog: A Prosocial Backbone for Conversational Agentscited by 109
  9. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 96
  10. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  11. 2024Learning to Decode Collaboratively with Multiple Language Modelscited by 62
  12. 2022Questions Are All You Need to Train a Dense Passage Retrievercited by 47
  13. 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 46
  14. 2024Quantifying the Gain in Weak-to-Strong Generalizationcited by 41
  15. 2023Small Language Models Fine-tuned to Coordinate Larger Language Models improve Complex Reasoningcited by 16
  16. 2024Improving Weak-to-Strong Generalization with Reliability-Aware Alignmentcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discoverycited by 23
  2. 2026Allspark: Weak to Strong Transfer via Alternating Chain of Thoughtcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model
  2. 2024Your Weak LLM is Secretly a Strong Teacher for Alignment
  3. 2026Local-First Intelligence via Proactive SLM Collaborating with Feedback-Providing LLM
  4. 2024Weak-to-Strong Reasoning
  5. 2026Compositional RL: Evaluating and Training LLMs for Collaborative Reasoning
  6. 2025Efficient LLM Collaboration via Planning
  7. 2026From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging
  8. 2026COMED: The Missing Middle Between Routing and Collaboration in Multi-LLM Inference