# Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling

arXiv: https://arxiv.org/abs/2410.03777 · HTML: https://mlsanity.com/paper/2410.03777/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 33 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 30 of 30)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [Deep Reinforcement Learning for Dialogue Generation](https://mlsanity.com/paper/1606.01541/) (2016)
- [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://mlsanity.com/paper/2309.11235/) (2023)
- [Evolutionary Optimization of Model Merging Recipes](https://mlsanity.com/paper/2403.13187/) (2024)
- [Large Language Model Routing with Benchmark Datasets](https://mlsanity.com/paper/2309.15789/) (2023)
- [Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models](https://mlsanity.com/paper/2311.08692/) (2023)
- [Knowledge Fusion of Large Language Models](https://mlsanity.com/paper/2401.10491/) (2024)
- [Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration](https://mlsanity.com/paper/2404.12715/) (2024)
- [FuseChat: Knowledge Fusion of Chat Models](https://mlsanity.com/paper/2408.07990/) (2024)
- [Fusing Models with Complementary Expertise](https://mlsanity.com/paper/2310.01542/) (2023)
- [Bridging the Gap between Different Vocabularies for LLM Ensemble](https://mlsanity.com/paper/2404.09492/) (2024)
- [Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling](https://mlsanity.com/paper/2406.12585/) (2024)
- [DisWOT: Student Architecture Search for Distillation WithOut Training](https://mlsanity.com/paper/2303.15678/) (2023)
- [Cool-Fusion: Fuse Large Language Models without Training](https://mlsanity.com/paper/2407.19807/) (2024)
- [A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models](https://mlsanity.com/paper/2305.18239/) (2023)

## Built on by (most cited first, 28 of 28)

- [Harnessing Multiple Large Language Models: A Survey on LLM Ensemble](https://mlsanity.com/paper/2502.18036/) (2025)
- [Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging](https://mlsanity.com/paper/2503.20641/) (2025)
- [Atom of Thoughts for Markov LLM Test-Time Scaling](https://mlsanity.com/paper/2502.12018/) (2025)
- [MergeBench: A Benchmark for Merging Domain-Specialized LLMs](https://mlsanity.com/paper/2505.10833/) (2025)
- [Fast Large Language Model Collaborative Decoding via Speculation](https://mlsanity.com/paper/2502.01662/) (2025)
- [Harnessing Consistency for Robust Test-Time LLM Ensemble](https://mlsanity.com/paper/2510.13855/) (2025)
- [LLMBoost: Make Large Language Models Stronger with Boosting](https://mlsanity.com/paper/2512.22309/) (2025)
- [Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization](https://mlsanity.com/paper/2607.22334/) (2026)
- [LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Merging](https://mlsanity.com/paper/2502.10749/) (2025)
- [RLAE: Reinforcement Learning-Assisted Ensemble for LLMs](https://mlsanity.com/paper/2506.00439/) (2025)
- [SpecFuse: Ensembling Large Language Models via Next-Segment Prediction](https://mlsanity.com/paper/2412.07380/) (2024)
- [AdaFuse: Adaptive Ensemble Decoding with Test-Time Scaling for LLMs](https://mlsanity.com/paper/2601.06022/) (2026)
- [When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM Ensembling](https://mlsanity.com/paper/2510.15346/) (2025)
- [Ensembling Language Models with Sequential Monte Carlo](https://mlsanity.com/paper/2603.05432/) (2026)
- [Dynamic Optimizations of LLM Ensembles with Two-Stage Reinforcement Learning Agents](https://mlsanity.com/paper/2502.04492/) (2025)
- [Dynamic Collaboration of Multi-Language Models based on Minimal Complete Semantic Units](https://mlsanity.com/paper/2508.18763/) (2025)
- [Lossless Vocabulary Reduction for Auto-Regressive Language Models](https://mlsanity.com/paper/2510.08102/) (2025)
- [Can Heterogeneous Language Models Be Fused?](https://mlsanity.com/paper/2604.01674/) (2026)
- [Tandem: Riding Together with Large and Small Language Models for Efficient Reasoning](https://mlsanity.com/paper/2604.23623/) (2026)
- [Inhibitory Attacks on Backdoor-based Fingerprinting for Large Language Models](https://mlsanity.com/paper/2601.04261/) (2026)
- [Rethinking LLM Ensembling from the Perspective of Mixture Models](https://mlsanity.com/paper/2605.00419/) (2026)
- [One for All: Update Parameterized Knowledge Across Multiple Models](https://mlsanity.com/paper/2506.00817/) (2025)
- [Training-Free Adaptation of New-Generation LLMs using Legacy Clinical Models](https://mlsanity.com/paper/2601.03423/) (2026)
- [SpecFed: Accelerating Federated LLM Inference with Speculative Decoding and Compressed Transmission](https://mlsanity.com/paper/2604.25777/) (2026)
- [DLLG: Dynamic Logit-Level Gating of LLM Experts](https://mlsanity.com/paper/2606.04378/) (2026)
- [Who Should Lead Decoding Now? Tracking Reliable Trajectories for Ensembling Masked Diffusion Language Models](https://mlsanity.com/paper/2606.16281/) (2026)
- [Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes](https://mlsanity.com/paper/2607.26627/) (2026)
- [Colla-Q: Toward Collaborative Experts in MoE Quantization via Minimax Precision Balancing](https://mlsanity.com/paper/2609.18131/) (2026)

## Abstract (arXiv)

Large language models (LLMs) exhibit varying strengths and weaknesses across different tasks, prompting recent studies to explore the benefits of ensembling models to leverage their complementary advantages. However, existing LLM ensembling methods often overlook model compatibility and struggle with inefficient alignment of probabilities across the entire vocabulary. In this study, we empirically investigate the factors influencing ensemble performance, identifying model performance, vocabulary size, and response style as key determinants, revealing that compatibility among models is essential for effective ensembling. This analysis leads to the development of a simple yet effective model selection strategy that identifies compatible models. Additionally, we introduce the textscUnion textscTop-k textscEnsembling (textscUniTE), a novel approach that efficiently combines models by focusing on the union of the top-k tokens from each model, thereby avoiding the need for full vocabulary alignment and reducing computational overhead. Extensive evaluations across multiple benchmarks demonstrate that textscUniTE significantly enhances performance compared to existing methods, offering a more efficient framework for LLM ensembling.