# Bridging the Gap between Different Vocabularies for LLM Ensemble

arXiv: https://arxiv.org/abs/2404.09492 · HTML: https://mlsanity.com/paper/2404.09492/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2024.

## Built on by (most cited first, 30 of 45)

- [Harnessing Multiple Large Language Models: A Survey on LLM Ensemble](https://mlsanity.com/paper/2502.18036/) (2025)
- [Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration](https://mlsanity.com/paper/2404.12715/) (2024)
- [Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models](https://mlsanity.com/paper/2407.06089/) (2024)
- [Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning](https://mlsanity.com/paper/2412.15797/) (2024)
- [Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling](https://mlsanity.com/paper/2406.12585/) (2024)
- [Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling](https://mlsanity.com/paper/2410.03777/) (2024)
- [Cool-Fusion: Fuse Large Language Models without Training](https://mlsanity.com/paper/2407.19807/) (2024)
- [TokAlign: Efficient Vocabulary Adaptation via Token Alignment](https://mlsanity.com/paper/2506.03523/) (2025)
- [Harnessing Consistency for Robust Test-Time LLM Ensemble](https://mlsanity.com/paper/2510.13855/) (2025)
- [Ensemble Learning for Large Language Models in Text and Code Generation: A Survey](https://mlsanity.com/paper/2503.13505/) (2025)
- [Hit the Sweet Spot! Span-Level Ensemble for Large Language Models](https://mlsanity.com/paper/2409.18583/) (2024)
- [LLMBoost: Make Large Language Models Stronger with Boosting](https://mlsanity.com/paper/2512.22309/) (2025)
- [Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization](https://mlsanity.com/paper/2607.22334/) (2026)
- [Sampling from Your Language Model One Byte at a Time](https://mlsanity.com/paper/2506.14123/) (2025)
- [SpecFuse: Ensembling Large Language Models via Next-Segment Prediction](https://mlsanity.com/paper/2412.07380/) (2024)
- [FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion](https://mlsanity.com/paper/2503.04222/) (2025)
- [Think in Parallel, Answer as One: Logit Averaging for Open-Ended Reasoning](https://mlsanity.com/paper/2512.02874/) (2025)
- [Diver: Large Language Model Decoding with Span-Level Mutual Information Verification](https://mlsanity.com/paper/2406.02120/) (2024)
- [DEPT: Decoupled Embeddings for Pre-training Language Models](https://mlsanity.com/paper/2410.05021/) (2024)
- [Position: Enough of Scaling LLMs! Lets Focus on Downscaling](https://mlsanity.com/paper/2505.00985/) (2025)
- [Representation Consistency for Accurate and Coherent LLM Answer Aggregation](https://mlsanity.com/paper/2506.21590/) (2025)
- [Speculating LLMs' Chinese Training Data Pollution from Their Tokens](https://mlsanity.com/paper/2508.17771/) (2025)
- [DFPE: A Diverse Fingerprint Ensemble for Enhancing LLM Performance](https://mlsanity.com/paper/2501.17479/) (2025)
- [Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling](https://mlsanity.com/paper/2503.19123/) (2025)
- [Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Process](https://mlsanity.com/paper/2512.23213/) (2025)
- [Token-level Ensembling of Models with Different Vocabularies](https://mlsanity.com/paper/2502.21265/) (2025)
- [GReaTER: Generate Realistic Tabular data after data Enhancement and Reduction](https://mlsanity.com/paper/2503.15564/) (2025)
- [MArgE: Meshing Argumentative Evidence from Multiple Large Language Models for Justifiable Claim Verification](https://mlsanity.com/paper/2508.02584/) (2025)
- [When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM Ensembling](https://mlsanity.com/paper/2510.15346/) (2025)
- [Ensembling Language Models with Sequential Monte Carlo](https://mlsanity.com/paper/2603.05432/) (2026)

## Abstract (arXiv)

Ensembling different large language models (LLMs) to unleash their complementary potential and harness their individual strengths is highly valuable. Nevertheless, vocabulary discrepancies among various LLMs have constrained previous studies to either selecting or blending completely generated outputs. This limitation hinders the dynamic correction and enhancement of outputs during the generation process, resulting in a limited capacity for effective ensemble. To address this issue, we propose a novel method to Ensemble LLMs via Vocabulary Alignment (EVA). EVA bridges the lexical gap among various LLMs, enabling meticulous ensemble at each generation step. Specifically, we first learn mappings between the vocabularies of different LLMs with the assistance of overlapping tokens. Subsequently, these mappings are employed to project output distributions of LLMs into a unified space, facilitating a fine-grained ensemble. Finally, we design a filtering strategy to exclude models that generate unfaithful tokens. Experimental results on commonsense reasoning, arithmetic reasoning, machine translation, and data-to-text generation tasks demonstrate the superiority of our approach compared with individual LLMs and previous ensemble methods conducted on complete outputs. Further analyses confirm that our approach can leverage knowledge from different language models and yield consistent improvement.