mlsanityData to 14 Oct 2026

Identifying Influential N-grams in Confidence Calibration via Regression Analysis

arXiv 2604.05757 ↗First posted 7 Apr 2026cs.CLcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

While large language models (LLMs) improve performance by explicit reasoning, their responses are often overconfident, even though they include linguistic expressions demonstrating uncertainty. In this work, we identify what linguistic expressions are related to confidence by applying the regression method. Specifically, we predict confidence of those linguistic expressions in the reasoning parts of LLMs as the dependent variables and analyze the relationship between a specific n-gram and confidence. Across multiple models and QA benchmarks, we show that LLMs remain overconfident when reasoning is involved and attribute this behavior to specific linguistic information. Interestingly, several of the extracted expressions coincide with cue phrases intentionally inserted on test-time scaling to improve reasoning performance. Through our test on causality and verification that the extracted linguistic information truly affects confidence, we reveal that confidence calibration is possible by simply suppressing those overconfident expressions without drops in performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 67 of its 85 references that are in the map. First posted on arXiv 7 Apr 2026.

Builds on

67 of 85 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025gpt-oss-120b & gpt-oss-20b Model Card8 Aug
  2. 2025Llama-Nemotron: Efficient Reasoning Models2 May
  3. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?31 Mar
  4. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey20 Mar
  5. 2025Confidence Improves Self-Consistency in LLMs10 Feb
  6. 2025s1: Simple test-time scaling31 Jan
  7. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems12 Dec
  8. 2022Language Models (Mostly) Know What They Know11 Jul

Other references · 59

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  11. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  12. 2017On Calibration of Modern Neural Networkscited by 4,960
38 more
  1. 2024Qwen2.5 Technical Reportcited by 4,942
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2016Concrete Problems in AI Safetycited by 1,642
  10. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  11. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  12. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  13. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  14. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  15. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  16. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  17. 2024Phi-4 Technical Reportcited by 711
  18. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  19. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  20. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  21. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  22. 2019Measuring Calibration in Deep Learningcited by 446
  23. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  24. 2021Unsolved Problems in ML Safetycited by 292
  25. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 285
  26. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 274
  27. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  28. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  29. 2023Med-HALT: Medical Domain Hallucination Test for Large Language Modelscited by 157
  30. 2025Phi-4-reasoning Technical Reportcited by 151
  31. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 150
  32. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  33. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 89
  34. 2023PMC-LLaMA: Towards Building Open-source Language Models for Medicinecited by 88
  35. 2023Self-Evaluation Improves Selective Generation in Large Language Modelscited by 76
  36. 2025A Survey of Uncertainty Estimation Methods on Large Language Modelscited by 62
  37. 2024Confidence Regulation Neurons in Language Modelscited by 57
  38. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48

50 most cited of 59 All 59 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Closing the Confidence-Faithfulness Gap in Large Language Models
  2. 2024Confidence in the Reasoning of Large Language Models
  3. 2026Quantifying Faithful Confidence Expression in Large Reasoning Models
  4. 2026Rethinking Uncertainty Evaluation in Large Language Models
  5. 2026Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models
  6. 2026Retrieval-Augmented Linguistic Calibration
  7. 2024Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models
  8. 2025Calibrating Verbalized Confidence with Self-Generated Distractors