mlsanityData to 14 Oct 2026

Popular but Wrong: Understanding and Mitigating LLM Overconfidence through Knowledge Popularity

arXiv 2505.17537 ↗First posted 23 May 2025cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) often produce incorrect answers with high confidence, yet the factors associated with such overconfidence remain insufficiently understood. We study this problem through the lens of knowledge popularity. Using entity-centric factual QA derived from Wikidata triplets, we characterize popularity through question entity popularity, answer popularity, and question-answer co-occurrence. We find two consistent patterns. First, hallucinated answers are far from random: compared with ground-truth answers, they tend to be more popular or more frequently associated with the question entity. Second, confidence is strongly tied to the popularity of generated answers: even among incorrect predictions, more popular answers or those with higher question-answer co-occurrence receive higher confidence. Together, these findings suggest that popular but wrong alternatives may contribute to overconfidence. We further show that popularity-related signals can mitigate overconfidence and improve overall confidence estimation. Across six models and three datasets, incorporating knowledge popularity reduces average confidence on incorrect answers from 0.765 to 0.254 and overall ECE from 0.356 to 0.050, while improving Alignment from 77.08% to 83.72%.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 18 of its 34 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

18 of 34 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2017On Calibration of Modern Neural Networkscited by 4,960
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  7. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  9. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  10. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  11. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  12. 2020Calibration of Pre-trained Transformerscited by 315
6 more
  1. 2022Prompting GPT-3 To Be Reliablecited by 260
  2. 2023SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistencycited by 90
  3. 2023Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Modelscited by 67
  4. 2024When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentationcited by 59
  5. 2025Towards Fully Exploiting LLM Internal States to Enhance Knowledge Boundary Perceptioncited by 27
  6. 2024Are Large Language Models More Honest in Their Probabilistic or Verbalized Confidence?cited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Do LVLMs Know What They Know? A Systematic Study of Knowledge Boundary Perception in LVLMscited by 4
  2. 2025Injecting External Knowledge into the Reasoning Process Enhances Retrieval-Augmented Generationcited by 3
  3. 2026AuthorityBench: Benchmarking LLM Authority Perception for Reliable Retrieval-Augmented Generationcited by 1
  4. 2026Latent Knowledge as a Predictor of Fact Acquisition in Fine-Tuned Large Language Modelscited by 0
  5. 2026One Score, Two Decisions: Selective Prediction on the Rare-Disease Tailcited by 0
  6. 2026Large Language Models Systematically Favor Popular Options: Evidence and Mitigation Across MCQscited by 0