mlsanityData to 14 Oct 2026

The Majority is not always right: RL training for solution aggregation

arXiv 2509.06870 ↗First posted 8 Sep 2025cs.CLcited by 47

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Scaling up test-time compute, by generating multiple independent solutions and selecting or aggregating among them, has become a central paradigm for improving large language models (LLMs) on challenging reasoning tasks. While most prior work relies on simple majority voting or reward model ranking to aggregate solutions, these approaches may only yield limited benefits. In this work, we propose to learn aggregation as an explicit reasoning skill: given a set of candidate solutions, we train an aggregator model to review, reconcile, and synthesize a final, correct answer using reinforcement learning from verifiable rewards. A key ingredient is careful balancing of easy and hard training examples, allowing the model to learn both to recover minority-but-correct answers as well as easy majority-correct answers. Empirically, we find our method, AggLM, outperforms both strong rule-based and reward-model baselines, across multiple benchmarks. Furthermore, it generalizes effectively to solutions from differing models, including stronger ones than contained in the training data, all while requiring substantially fewer tokens than majority voting with larger numbers of solutions.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 16 of its 21 references that are in the map. First posted on arXiv 8 Sep 2025.

Builds on

16 of 21 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan

Other references · 15

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  5. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  6. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  7. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 290
  8. 2025Deep Think with Confidencecited by 205
  9. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  10. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMscited by 115
  11. 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
  12. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 77
3 more
  1. 2025Learning to Reason Across Parallel Samples for LLM Reasoningcited by 32
  2. 2024Mirror-Consistency: Harnessing Inconsistency in Majority Votingcited by 17
  3. 2022Uncertainty Determines the Adequacy of the Mode and the Tractability of Decoding in Sequence-to-Sequence Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 30.

47 papers.

Sort:

By area:

  1. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 45
  2. 2025Recursive Self-Aggregation Unlocks Deep Thinking in Large Language Modelscited by 35
  3. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 27
  4. 2025Rethinking Thinking Tokens: LLMs as Improvement Operatorscited by 21
  5. 2026TTCS: Test-Time Curriculum Synthesis for Self-Evolvingcited by 19
  6. 2026Demystifying Multi-Agent Debate: The Role of Confidence and Diversitycited by 18
  7. 2026Agentic Aggregation for Parallel Scaling of Long-Horizon Agentic Taskscited by 18
  8. 2026Autodata: An agentic data scientist to create high quality synthetic datacited by 16
  9. 2026Scaling Test-Time Compute for Agentic Codingcited by 15
  10. 2026V1: Unifying Generation and Self-Verification for Parallel Reasonerscited by 12
  11. 2026LLMs Improving LLMs: Agentic Discovery for Test-Time Scalingcited by 10
  12. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
35 more
  1. 2025Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculationcited by 8
  2. 2025The Era of Agentic Organization: Learning to Organize with Language Modelscited by 7
  3. 2025Making, not Taking, the Best of Ncited by 6
  4. 2025Generalized Parallel Scaling with Interdependent Generationscited by 6
  5. 2025Think in Parallel, Answer as One: Logit Averaging for Open-Ended Reasoningcited by 6
  6. 2026MoCo: A One-Stop Shop for Model Collaboration Researchcited by 6
  7. 2026Epistemic Context Learning: Building Trust the Right Way in LLM-Based Multi-Agent Systemscited by 6
  8. 2026Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoningcited by 6
  9. 2025Off-Trajectory Reasoning: Can LLMs Collaborate on Reasoning Trajectory?cited by 5
  10. 2025Plantain: Plan-Answer Interleaved Reasoningcited by 5
  11. 2026Understanding Performance Gap Between Parallel and Sequential Sampling in Large Reasoning Modelscited by 4
  12. 2025Best-of-∞ -- Asymptotic Performance of Test-Time LLM Ensemblingcited by 3
  13. 2025TrajSelector: Harnessing Latent Representations for Efficient and Effective Best-of-N in Large Reasoning Modelcited by 3
  14. 2025The Law of Multi-Model Collaboration: Scaling Limits of Model Ensembling for Large Language Modelscited by 3
  15. 2026Reject, Resample, Repeat: Understanding Parallel Reasoning in Language Model Inferencecited by 3
  16. 2026Learning Adaptive LLM Decodingcited by 3
  17. 2026Reasoning over mathematical objects: on-policy reward modeling and test time aggregationcited by 3
  18. 2026AgentFugue: Agent Scaling for Long-Horizon Tasks through Collective Reasoningcited by 3
  19. 2026Multi-Agent Computer Usecited by 3
  20. 2026Learning Generative Selection for Best-of-Ncited by 2
  21. 2026The Single-Multi Evolution Loop for Self-Improving Model Collaboration Systemscited by 2
  22. 2026Dual Consensus: Escaping from Spurious Majority in Unsupervised RLVR via Two-Stage Vote Mechanismcited by 2
  23. 2026CAPS: Cascaded Adaptive Pairwise Selection for Efficient Parallel Reasoningcited by 2
  24. 2026Scaling Participation in Modular AI Systemscited by 2
  25. 2025Critique to Verify: Accurate and Honest Test-Time Scaling with RL-Trained Verifierscited by 1
  26. 2026Learning from Self-Debate: Preparing Reasoning Models for Multi-Agent Debatecited by 1
  27. 2026Divide-and-Conquer CoT: RL for Reducing Latency via Parallel Reasoningcited by 1
  28. 2026One-Token Verification for Reasoning Correctness Estimationcited by 1
  29. 2026FineVerify: Scaling Test-Time Compute with Fine-Grained Self-Verification for Agentic Searchcited by 1
  30. 2025Evolutionary Profiles for Protein Fitness Predictioncited by 0
  31. 2026Efficient Reasoning on the Edgecited by 0
  32. 2026SCATR: Simple Calibrated Test-Time Rankingcited by 0
  33. 2026Performance Foundations of Parallel & Distributed Reasoning Language Modelscited by 0
  34. 2026Multi-LLM Collaborative Alignment via Stackelberg Gamescited by 0
  35. 2026Compositional RL: Evaluating and Training LLMs for Collaborative Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Models
  2. 2026Divergent-Convergent Reasoning: Scaling Test-Time Compute through Structured Solution Synthesis
  3. 2026DARL: Encouraging Diverse Answers for General Reasoning without Verifiers
  4. 2025Self-Improvement of Language Models by Post-Training on Multi-Agent Debate
  5. 2025From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization
  6. 2026Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPO
  7. 2026A Single Revision Step Improves Token-Efficient LLM Reasoning
  8. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoning