mlsanityData to 14 Oct 2026

Activation-Guided Consensus Merging for Large Language Models

arXiv 2505.14009 ↗First posted 20 May 2025cs.CLcited by 12

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Recent research has increasingly focused on reconciling the reasoning capabilities of System 2 with the efficiency of System 1. While existing training-based and prompt-based approaches face significant challenges in terms of efficiency and stability, model merging emerges as a promising strategy to integrate the diverse capabilities of different Large Language Models (LLMs) into a unified model. However, conventional model merging methods often assume uniform importance across layers, overlooking the functional heterogeneity inherent in neural components. To address this limitation, we propose Activation-Guided Consensus Merging (ACM), a plug-and-play merging framework that determines layer-specific merging coefficients based on mutual information between activations of pre-trained and fine-tuned models. ACM effectively preserves task-specific capabilities without requiring gradient computations or additional training. Extensive experiments on Long-to-Short (L2S) and general merging tasks demonstrate that ACM consistently outperforms all baseline methods. For instance, in the case of Qwen-7B models, TIES-Merging equipped with ACM achieves a 55.3% reduction in response length while simultaneously improving reasoning accuracy by 1.3 points.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 37 of its 46 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

37 of 46 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning2 Apr
  2. 2025Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging26 Mar
  3. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning6 Mar
  4. 2025DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Models6 Mar
  5. 2025Self-Training Elicits Concise Reasoning in Large Language Models27 Feb
  6. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMs17 Feb
  7. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuning13 Feb
  8. 2025s1: Simple test-time scaling31 Jan
  9. 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning22 Jan
  10. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM12 Mar

Other references · 27

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  7. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  9. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  10. 2022Editing Models with Task Arithmeticcited by 1,120
  11. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  12. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
15 more
  1. 2025LIMO: Less is More for Reasoningcited by 498
  2. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  3. 2024Distilling System 2 into System 1cited by 159
  4. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  5. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 116
  6. 2019A Mutual Information Maximization Perspective of Language Representation Learningcited by 85
  7. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  8. 2024HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generationcited by 39
  9. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  10. 2023Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edgecited by 33
  11. 2025Activation-Informed Merging of Large Language Modelscited by 30
  12. 2024Learning to Maximize Mutual Information for Chain-of-Thought Distillationcited by 23
  13. 2025Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Modelscited by 16
  14. 2025LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Mergingcited by 7
  15. 20251bit-Merging: Dynamic Quantized Merging for Large Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 10.

12 papers.

  1. 2025TL;DR: Too Long, Do Re-weighting for Efficient LLM Reasoning Compressioncited by 16
  2. 2025From Efficiency to Adaptivity: A Deeper Look at Adaptive Reasoning in Large Language Modelscited by 12
  3. 2026RAIN-Merging: A Gradient-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Formatcited by 9
  4. 2026Label-Free Cross-Task LoRA Merging with Null-Space Compressioncited by 4
  5. 2026System 1&2 Synergy via Dynamic Model Interpolationcited by 3
  6. 2026Sparsity Curse: Understanding RLVR Model Parameter Space from Model Mergingcited by 2
  7. 2026QA-Merging: Query-Adaptive Reasoning via Layer Selective Model Mergingcited by 1
  8. 2026Merging Beyond: Streaming LLM Updates via Activation-Guided Rotationscited by 1
  9. 2026From Parameters to Behaviors: A Survey of Model Fusion for Large Language Modelscited by 1
  10. 2026Data-Free Layer-Adaptive Merging via Fisher Information for Long-to-Short Reasoning LLMscited by 0
  11. 2026Saliency-Aware Model Mergingcited by 0
  12. 2026Towards Efficient Reasoning in LLM-Based Recommender Systems via Model Mergingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions
  2. 2025Model Merging in Pre-training of Large Language Models
  3. 2025Fine, I'll Merge It Myself: A Multi-Fidelity Framework for Automated Model Merging
  4. 2024Unconstrained Model Merging for Enhanced LLM Reasoning
  5. 2024Channel Merging: Preserving Specialization for Merged Experts
  6. 2025Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and Uncertainty Based Routing
  7. 2026Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective
  8. 2026Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs