# TECP: Token-Entropy Conformal Prediction for LLMs

arXiv: https://arxiv.org/abs/2509.00461 · HTML: https://mlsanity.com/paper/2509.00461/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 30 Aug 2025.

## Builds on (most cited first, 22 of 22)

- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Conformal prediction beyond exchangeability](https://mlsanity.com/paper/2202.13415/) (2022)
- [Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models](https://mlsanity.com/paper/2307.01379/) (2023)
- [ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guarantees](https://mlsanity.com/paper/2407.00499/) (2024)
- [Detecting and Evaluating Medical Hallucinations in Large Vision Language Models](https://mlsanity.com/paper/2406.10185/) (2024)
- [ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM](https://mlsanity.com/paper/2506.14766/) (2025)
- [Backdoor Cleaning without External Guidance in MLLM Fine-tuning](https://mlsanity.com/paper/2505.16916/) (2025)
- [Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspective](https://mlsanity.com/paper/2411.14258/) (2024)
- [CoMT: Chain-of-Medical-Thought Reduces Hallucination in Medical Report Generation](https://mlsanity.com/paper/2406.11451/) (2024)
- [COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees](https://mlsanity.com/paper/2506.20178/) (2025)
- [SPOT! Revisiting Video-Language Models for Event Understanding](https://mlsanity.com/paper/2311.12919/) (2023)
- [Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Models](https://mlsanity.com/paper/2410.08174/) (2024)
- [Word-Sequence Entropy: Towards Uncertainty Estimation in Free-Form Medical Question Answering Applications and Beyond](https://mlsanity.com/paper/2402.14259/) (2024)
- [PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications](https://mlsanity.com/paper/2405.19266/) (2024)
- [Can LLMs' Tuning Methods Work in Medical Multimodal Domain?](https://mlsanity.com/paper/2403.06407/) (2024)
- [Conformal Prediction as Bayesian Quadrature](https://mlsanity.com/paper/2502.13228/) (2025)
- [Conformal Prediction with Cellwise Outliers: A Detect-then-Impute Approach](https://mlsanity.com/paper/2505.04986/) (2025)
- [MV-Debate: Multi-view Agent Debate with Dynamic Reflection Gating for Multimodal Harmful Content Detection in Social Media](https://mlsanity.com/paper/2508.05557/) (2025)

## Built on by (most cited first, 6 of 6)

- [Entropy Alone is Insufficient for Safe Selective Prediction in LLMs](https://mlsanity.com/paper/2603.21172/) (2026)
- [CAOS: Conformal Aggregation of One-Shot Predictors](https://mlsanity.com/paper/2601.05219/) (2026)
- [Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection](https://mlsanity.com/paper/2605.18512/) (2026)
- [When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction](https://mlsanity.com/paper/2608.15592/) (2026)
- [Solving versus Verifying: Catching Contradictions in Tax Reasoning Systems](https://mlsanity.com/paper/2609.05928/) (2026)
- [ConformalGuard: False-Positive-Controlled Action Gating for LM Classifiers](https://mlsanity.com/paper/2610.14365/) (2026)

## Abstract (arXiv)

Uncertainty quantification (UQ) for open-ended language generation remains a critical yet underexplored challenge, especially under black-box constraints where internal model signals are inaccessible. In this paper, we introduce Token-Entropy Conformal Prediction (TECP), a novel framework that leverages token-level entropy as a logit-free, reference-free uncertainty measure and integrates it into a split conformal prediction (CP) pipeline to construct prediction sets with formal coverage guarantees. Unlike existing approaches that rely on semantic consistency heuristics or white-box features, TECP directly estimates epistemic uncertainty from the token entropy structure of sampled generations and calibrates uncertainty thresholds via CP quantiles to ensure provable error control. Empirical evaluations across six large language models and two benchmarks (CoQA and TriviaQA) demonstrate that TECP consistently achieves reliable coverage and compact prediction sets, outperforming prior self-consistency-based UQ methods. Our method provides a principled and efficient solution for trustworthy generation in black-box LLM settings.