# Calibrating Reasoning in Language Models with Internal Consistency

arXiv: https://arxiv.org/abs/2405.18711 · HTML: https://mlsanity.com/paper/2405.18711/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 31 of its 57 references that are in the map.
- First posted on arXiv 29 May 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 31)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://mlsanity.com/paper/2012.13048/) (2020)
- [Faithful Chain-of-Thought Reasoning](https://mlsanity.com/paper/2301.13379/) (2023)
- [Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://mlsanity.com/paper/2404.02258/) (2024)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [Chain-of-Thought Reasoning Without Prompting](https://mlsanity.com/paper/2402.10200/) (2024)
- [A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity](https://mlsanity.com/paper/2401.01967/) (2024)
- [Implicit Chain of Thought Reasoning via Knowledge Distillation](https://mlsanity.com/paper/2311.01460/) (2023)
- [Do Large Language Models Latently Perform Multi-Hop Reasoning?](https://mlsanity.com/paper/2402.16837/) (2024)
- [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024)
- [Overthinking the Truth: Understanding how Language Models Process False Demonstrations](https://mlsanity.com/paper/2307.09476/) (2023)
- [Residual Connections Encourage Iterative Inference](https://mlsanity.com/paper/1710.04773/) (2017)
- [Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMs](https://mlsanity.com/paper/2404.15676/) (2024)
- [Thermometer: Towards Universal Calibration for Large Language Models](https://mlsanity.com/paper/2403.08819/) (2024)

## Built on by (most cited first, 28 of 28)

- [Internal Consistency and Self-Feedback in Large Language Models: A Survey](https://mlsanity.com/paper/2407.14507/) (2024)
- [InfAlign: Inference-aware language model alignment](https://mlsanity.com/paper/2412.19792/) (2024)
- [Towards Trustworthy AI: A Review of Ethical and Robust Large Language Models](https://mlsanity.com/paper/2407.13934/) (2024)
- [Thought calibration: Efficient and confident test-time scaling](https://mlsanity.com/paper/2505.18404/) (2025)
- [Graph-based Approaches and Functionalities in Retrieval-Augmented Generation: A Comprehensive Survey](https://mlsanity.com/paper/2504.10499/) (2025)
- [Unveiling Factual Recall Behaviors of Large Language Models through Knowledge Neurons](https://mlsanity.com/paper/2408.03247/) (2024)
- [Query-Level Uncertainty in Large Language Models](https://mlsanity.com/paper/2506.09669/) (2025)
- [Mining Intrinsic Rewards from LLM Hidden States for Efficient Best-of-N Sampling](https://mlsanity.com/paper/2505.12225/) (2025)
- [Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimation](https://mlsanity.com/paper/2512.22245/) (2025)
- [Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation](https://mlsanity.com/paper/2602.09305/) (2026)
- [Practical estimation of the optimal classification error with soft labels and calibration](https://mlsanity.com/paper/2505.20761/) (2025)
- [Slm-mux: Orchestrating small language models for reasoning](https://mlsanity.com/paper/2510.05077/) (2025)
- [None of the Above, Less of the Right: Parallel Patterns between Humans and LLMs on Multi-Choice Questions Answering](https://mlsanity.com/paper/2503.01550/) (2025)
- [ORCH: many analyses, one merge-a deterministic multi-agent orchestrator for discrete-choice reasoning with EMA-guided routing](https://mlsanity.com/paper/2602.01797/) (2026)
- [Online Reasoning Calibration: Test-Time Training Enables Generalizable Conformal LLM Reasoning](https://mlsanity.com/paper/2604.01170/) (2026)
- [EQUATOR: A Deterministic Framework for Evaluating LLM Reasoning with Open-Ended Questions. # v1.0.0-beta](https://mlsanity.com/paper/2501.00257/) (2024)
- [Latent Debate: A Surrogate Framework for Interpreting LLM Thinking](https://mlsanity.com/paper/2512.01909/) (2025)
- [Knowing the Facts but Choosing the Shortcut: Understanding How Large Language Models Compare Entities](https://mlsanity.com/paper/2510.16815/) (2025)
- [Benchmarking the rationality of AI decision making using the transitivity axiom](https://mlsanity.com/paper/2502.10554/) (2025)
- [Deep Hidden Cognition Facilitates Reliable Chain-of-Thought Reasoning](https://mlsanity.com/paper/2507.10007/) (2025)
- [Unlocking the Pre-Trained Model as a Dual-Alignment Calibrator for Post-Trained LLMs](https://mlsanity.com/paper/2601.04277/) (2026)
- [Prefix-Safe Bayesian Belief Tracking for LLM Reasoning Reliability:Separating Calibration from Ranking](https://mlsanity.com/paper/2605.27712/) (2026)
- [TrustDataFilter:Leveraging Trusted Knowledge Base Data for More Effective Filtering of Unknown Information](https://mlsanity.com/paper/2502.15714/) (2025)
- [Real-Time Progress Prediction in Reasoning Language Models](https://mlsanity.com/paper/2506.23274/) (2025)
- [Implicit Reasoning Steering via Concept Chaining](https://mlsanity.com/paper/2607.14242/) (2026)
- [CHAIN: Calibrated LLM Forecasting via Causal-Temporal Hypergraph Inference](https://mlsanity.com/paper/2609.36689/) (2026)
- [Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs](https://mlsanity.com/paper/2610.01428/) (2026)
- [Using Evidence-Centered Design to Guide Prompt Engineering in Measurement: An Illustration with Automated Essay Scoring](https://mlsanity.com/paper/2610.15231/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive capabilities in various reasoning tasks, aided by techniques like chain-of-thought prompting that elicits verbalized reasoning. However, LLMs often generate text with obvious mistakes and contradictions, raising doubts about their ability to robustly process and utilize generated rationales. In this work, we investigate reasoning in LLMs through the lens of internal representations, focusing on how these representations are influenced by generated rationales. Our preliminary analysis reveals that while generated rationales improve answer accuracy, inconsistencies emerge between the model's internal representations in middle layers and those in final layers, potentially undermining the reliability of their reasoning processes. To address this, we propose internal consistency as a measure of the model's confidence by examining the agreement of latent predictions decoded from intermediate layers. Extensive empirical studies across different models and datasets demonstrate that internal consistency effectively distinguishes between correct and incorrect reasoning paths. Motivated by this, we propose a new approach to calibrate reasoning by up-weighting reasoning paths with high internal consistency, resulting in a significant boost in reasoning performance. Further analysis uncovers distinct patterns in attention and feed-forward modules across layers, providing insights into the emergence of internal inconsistency. In summary, our results demonstrate the potential of using internal representations for self-evaluation of LLMs. Our code is available at github.com/zhxieml/internal-consistency.