mlsanityData to 7 Oct 2026

From 'May' to 'Is': Certainty Distortion in Language Model Rewriting

arXiv 2606.07951 ↗First posted 6 Jun 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Humans increasingly turn to Language Models (LMs) in ways that shape beliefs and drive decisions, including discussing, rewriting, and summarizing information from scientific articles, news, and medical reports. However, in these domains, where it often matters how confidently a claim is expressed, little is known about whether LMs faithfully preserve the degree of confidence. In this work, we investigate certainty distortion in LMs, defined as meaningful changes in expressed certainty during transformations intended to preserve meaning. We propose an LM-based evaluation metric that is consistent with population-level judgments of certainty. Using this metric, we characterize certainty distortion across different sizes and families of models in the context of scientific and medical communication tasks. Our results show that certainty distortion affects up to 75% of LM outputs and is systematically asymmetric in rewriting tasks with most LMs being 1.5-2x more likely to increase the expressed certainty than to decrease it. These effects can compound over repeated paraphrasing: in the medical domain, claude-haiku-4.5 increases certainty in 20% of examples after a single iteration, increasing to 40% after five iterations. Prompt-based interventions reduce overall certainty distortion but do not eliminate it. Together, these findings reveal a general bias toward inflating expressed certainty, with direct implications for users who rely on LMs in high-stakes domains.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 79 references that are in the map. First posted on arXiv 6 Jun 2026.

Builds on

33 of 79 references.

References · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  8. 2024A Survey on LLM-as-a-Judgecited by 1,384
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  10. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  11. 2023How Language Model Hallucinations Can Snowballcited by 310
  12. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
21 more
  1. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  2. 2024What Large Language Models Know and What People Think They Knowcited by 109
  3. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  4. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  5. 2024A Closer Look at Claim Decompositioncited by 40
  6. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 38
  7. 2024Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluationcited by 29
  8. 2021Measuring Sentence-Level and Aspect-Level (Un)certainty in Science Communicationscited by 27
  9. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Modelscited by 21
  10. 2026How LLMs Distort Our Written Languagecited by 20
  11. 2024When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settingscited by 17
  12. 2022Modeling Information Change in Science Communication with Semantically Matched Paraphrasescited by 14
  13. 2024MT-Ranker: Reference-free machine translation evaluation by inter-system rankingcited by 14
  14. 2025LLM as a Broken Telephone: Iterative Generation Distorts Informationcited by 12
  15. 2024Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluationcited by 9
  16. 2024Understanding Fine-grained Distortions in Reports of Scientific Findingscited by 8
  17. 2025Hallucinate at the Last in Long Response Generation: A Case Study on Long Document Summarizationcited by 6
  18. 2025How Does Response Length Affect Long-Form Factualitycited by 6
  19. 2025LEAF: Knowledge Distillation of Text Embedding Models with Teacher-Aligned Representationscited by 4
  20. 2025How AI Responses Shape User Beliefs: The Effects of Information Detail and Confidence on Belief Strength and Stancecited by 2
  21. 2025On Fact and Frequency: LLM Responses to Misinformation Expressed with Uncertaintycited by 1

Built on by

Not cited yet.