# CharED: Character-wise Ensemble Decoding for Large Language Models

arXiv: https://arxiv.org/abs/2407.11009 · HTML: https://mlsanity.com/paper/2407.11009/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 28 of its 31 references that are in the map.
- First posted on arXiv 25 Jun 2024.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 28 of 28)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015)
- [Character-Level Language Modeling with Deeper Self-Attention](https://mlsanity.com/paper/1808.04444/) (2018)
- [Tuning Language Models by Proxy](https://mlsanity.com/paper/2401.08565/) (2024)
- [Zero-Resource Translation with Multi-Lingual Neural Machine Translation](https://mlsanity.com/paper/1606.04164/) (2016)
- [What do tokens know about their characters and how do they know it?](https://mlsanity.com/paper/2206.02608/) (2022)
- [Simple Fusion: Return of the Language Model](https://mlsanity.com/paper/1809.00125/) (2018)
- [Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation](https://mlsanity.com/paper/2302.14220/) (2023)
- [Character-Level Language Modeling with Hierarchical Recurrent Neural Networks](https://mlsanity.com/paper/1609.03777/) (2016)
- [Can Character-based Language Models Improve Downstream Task Performance in Low-Resource and Noisy Language Scenarios?](https://mlsanity.com/paper/2110.13658/) (2021)
- [Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models](https://mlsanity.com/paper/2403.00417/) (2024)
- [Twist Decoding: Diverse Generators Guide Each Other](https://mlsanity.com/paper/2205.09273/) (2022)

## Built on by (most cited first, 3 of 3)

- [Evaluating the Prompt Steerability of Large Language Models](https://mlsanity.com/paper/2411.12405/) (2024)
- [Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles](https://mlsanity.com/paper/2410.09303/) (2024)
- [Inference-Time Consensus for Mitigating Hidden Behaviors from LLM Fine-Tuning](https://mlsanity.com/paper/2607.23394/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown remarkable potential for problem solving, with open source models achieving increasingly impressive performance on benchmarks measuring areas from logical reasoning to mathematical ability. Ensembling models can further improve capabilities across a variety of domains. However, conventional methods of combining models at inference time such as shallow fusion necessitate a shared vocabulary and tokenization, and alternatives like fine-tuning for domain-specific performance are both time consuming and computationally expensive. We therefore present an inference-time ensembling algorithm aimed at "averaging" outputs from multiple LLMs and illustrate its improved performance across multiple domains compared to its constituent models alone. Character-wise ensemble decoding, CharED, finds the marginal distribution of each character for an individual model and performs a weighted average to generate an output, character by character. In coding, math, and toxicity benchmarks, we find our proposed model able to combine complimentary strengths of multiple LLMs, regardless of vocabulary, tokenization, or model size.