# Improving Self Consistency in LLMs through Probabilistic Tokenization

arXiv: https://arxiv.org/abs/2407.03678 · HTML: https://mlsanity.com/paper/2407.03678/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 0 of its 44 references that are in the map.
- First posted on arXiv 4 Jul 2024.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Built on by (most cited first, 8 of 8)

- [Number Cookbook: Number Understanding of Language Models and How to Improve It](https://mlsanity.com/paper/2411.03766/) (2024)
- [Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models](https://mlsanity.com/paper/2502.11075/) (2025)
- [Revisiting Self-Consistency from Dynamic Distributional Alignment Perspective on Answer Aggregation](https://mlsanity.com/paper/2502.19830/) (2025)
- [Speculative Decoding for Multi-Sample Inference](https://mlsanity.com/paper/2503.05330/) (2025)
- [AdaFuse: Adaptive Ensemble Decoding with Test-Time Scaling for LLMs](https://mlsanity.com/paper/2601.06022/) (2026)
- [Scaling LLM Pre-training with Vocabulary Curriculum](https://mlsanity.com/paper/2502.17910/) (2025)
- [From Text to Insight: Leveraging Large Language Models for Performance Evaluation in Management](https://mlsanity.com/paper/2408.05328/) (2024)
- [GEMMA-SQL: A Novel Text-to-SQL Model Based on Large Language Models](https://mlsanity.com/paper/2511.04710/) (2025)

## Abstract (arXiv)

Prior research has demonstrated noticeable performance gains through the use of probabilistic tokenizations, an approach that involves employing multiple tokenizations of the same input string during the training phase of a language model. Despite these promising findings, modern large language models (LLMs) have yet to be trained using probabilistic tokenizations. Interestingly, while the tokenizers of these contemporary LLMs have the capability to generate multiple tokenizations, this property remains underutilized. In this work, we propose a novel method to leverage the multiple tokenization capabilities of modern LLM tokenizers, aiming to enhance the self-consistency of LLMs in reasoning tasks. Our experiments indicate that when utilizing probabilistic tokenizations, LLMs generate logically diverse reasoning paths, moving beyond mere surface-level linguistic diversity.We carefully study probabilistic tokenization and offer insights to explain the self consistency improvements it brings through extensive experimentation on 5 LLM families and 4 reasoning benchmarks.