# From 'May' to 'Is': Certainty Distortion in Language Model Rewriting

arXiv: https://arxiv.org/abs/2606.07951 · HTML: https://mlsanity.com/paper/2606.07951/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 79 references that are in the map.
- First posted on arXiv 6 Jun 2026.

## Builds on (most cited first, 30 of 33)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [How Language Model Hallucinations Can Snowball](https://mlsanity.com/paper/2305.13534/) (2023)
- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [What Large Language Models Know and What People Think They Know](https://mlsanity.com/paper/2401.13835/) (2024)
- [Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty](https://mlsanity.com/paper/2401.06730/) (2024)
- [Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models](https://mlsanity.com/paper/2302.13439/) (2023)
- [A Closer Look at Claim Decomposition](https://mlsanity.com/paper/2403.11903/) (2024)
- [Evaluation of Best-of-N Sampling Strategies for Language Model Alignment](https://mlsanity.com/paper/2502.12668/) (2025)
- [Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation](https://mlsanity.com/paper/2410.20774/) (2024)
- [Measuring Sentence-Level and Aspect-Level (Un)certainty in Science Communications](https://mlsanity.com/paper/2109.14776/) (2021)
- [A Probabilistic Perspective on Unlearning and Alignment for Large Language Models](https://mlsanity.com/paper/2410.03523/) (2024)
- [How LLMs Distort Our Written Language](https://mlsanity.com/paper/2603.18161/) (2026)
- [When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settings](https://mlsanity.com/paper/2407.04503/) (2024)
- [Modeling Information Change in Science Communication with Semantically Matched Paraphrases](https://mlsanity.com/paper/2210.13001/) (2022)
- [MT-Ranker: Reference-free machine translation evaluation by inter-system ranking](https://mlsanity.com/paper/2401.17099/) (2024)
- [LLM as a Broken Telephone: Iterative Generation Distorts Information](https://mlsanity.com/paper/2502.20258/) (2025)
- [Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation](https://mlsanity.com/paper/2411.16638/) (2024)
- [Understanding Fine-grained Distortions in Reports of Scientific Findings](https://mlsanity.com/paper/2402.12431/) (2024)
- [Hallucinate at the Last in Long Response Generation: A Case Study on Long Document Summarization](https://mlsanity.com/paper/2505.15291/) (2025)
- [How Does Response Length Affect Long-Form Factuality](https://mlsanity.com/paper/2505.23295/) (2025)

## Abstract (arXiv)

Humans increasingly turn to Language Models (LMs) in ways that shape beliefs and drive decisions, including discussing, rewriting, and summarizing information from scientific articles, news, and medical reports. However, in these domains, where it often matters how confidently a claim is expressed, little is known about whether LMs faithfully preserve the degree of confidence. In this work, we investigate certainty distortion in LMs, defined as meaningful changes in expressed certainty during transformations intended to preserve meaning. We propose an LM-based evaluation metric that is consistent with population-level judgments of certainty. Using this metric, we characterize certainty distortion across different sizes and families of models in the context of scientific and medical communication tasks. Our results show that certainty distortion affects up to 75% of LM outputs and is systematically asymmetric in rewriting tasks with most LMs being 1.5-2x more likely to increase the expressed certainty than to decrease it. These effects can compound over repeated paraphrasing: in the medical domain, claude-haiku-4.5 increases certainty in 20% of examples after a single iteration, increasing to 40% after five iterations. Prompt-based interventions reduce overall certainty distortion but do not eliminate it. Together, these findings reveal a general bias toward inflating expressed certainty, with direct implications for users who rely on LMs in high-stakes domains.