mlsanityData to 7 Oct 2026

Uncertainty is Fragile: Manipulating Uncertainty in Large Language Models

arXiv 2407.11282 ↗First posted 15 Jul 2024cs.CLcited by 13

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Large Language Models (LLMs) are employed across various high-stakes domains, where the reliability of their outputs is crucial. One commonly used method to assess the reliability of LLMs' responses is uncertainty estimation, which gauges the likelihood of their answers being correct. While many studies focus on improving the accuracy of uncertainty estimations for LLMs, our research investigates the fragility of uncertainty estimation and explores potential attacks. We demonstrate that an attacker can embed a backdoor in LLMs, which, when activated by a specific trigger in the input, manipulates the model's uncertainty without affecting the final output. Specifically, the proposed backdoor attack method can alter an LLM's output probability distribution, causing the probability distribution to converge towards an attacker-predefined distribution while ensuring that the top-1 prediction remains unchanged. Our experimental results demonstrate that this attack effectively undermines the model's self-evaluation reliability in multiple-choice questions. For instance, we achieved a 100 attack success rate (ASR) across three different triggering strategies in four models. Further, we investigate whether this manipulation generalizes across different prompts and domains. This work highlights a significant threat to the reliability of LLMs and underscores the need for future defenses against such attacks. The code is available at https://github.com/qcznlp/uncertainty_attack.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 48 of its 69 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

48 of 69 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024TrojFM: Resource-efficient Backdoor Attacks against Very Large Foundation Models27 May
  2. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models20 Jan
  3. 2023PoisonPrompt: Backdoor Attack on Prompt-based Large Language Models19 Oct
  4. 2023Backdoor Attacks for In-Context Learning with Language Models27 Jul
  5. 2023NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Models28 May
  6. 2022Rethinking the Reverse-engineering of Trojan Triggers27 Oct

Other references · 42

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  7. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  8. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  9. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  10. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  11. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  12. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
30 more
  1. 2019Hidden Trigger Backdoor Attackscited by 359
  2. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  3. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  4. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  5. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  6. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  7. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  8. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  9. 2020Clean-Label Backdoor Attacks on Video Recognition Modelscited by 162
  10. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  11. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 121
  12. 2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 103
  13. 2022BppAttack: Stealthy and Efficient Trojan Attacks against Deep Neural Networks via Image Quantization and Contrastive Adversarial Learningcited by 82
  14. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 81
  15. 2018Quantifying Uncertainties in Natural Language Processing Taskscited by 79
  16. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 78
  17. 2023MathChat: Converse to Tackle Challenging Math Problems with LLM Agentscited by 67
  18. 2022Exploring the Universal Vulnerability of Prompt-based Learning Paradigmcited by 56
  19. 2024Determinants of LLM-assisted Decision-Makingcited by 55
  20. 2023UNICORN: A Unified Backdoor Trigger Inversion Frameworkcited by 44
  21. 2023UP5: Unbiased Foundation Model for Fairness-aware Recommendationcited by 38
  22. 2023LLM Performance Predictors are good initializers for Architecture Searchcited by 29
  23. 2023TrojText: Test-time Invisible Textual Trojan Insertioncited by 26
  24. 2023Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Settingcited by 20
  25. 2024DeLLMa: Decision Making Under Uncertainty with Large Language Modelscited by 20
  26. 2024Health-LLM: Personalized Retrieval-Augmented Disease Prediction Systemcited by 17
  27. 2024Probabilities of Chat LLMs Are Miscalibrated but Still Predict Correctness on Multiple-Choice Q&Acited by 16
  28. 2021You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradientcited by 14
  29. 2024Invisible Backdoor Attack against Self-supervised Learningcited by 5
  30. 2022Discover, Explanation, Improvement: An Automatic Slice Detection Framework for Natural Language Processingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 4.

13 papers.

Sort:

By area:

  1. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 146
  2. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  3. 2025Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Modelscited by 61
  4. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 14
  5. 2025Position: Towards a Responsible LLM-empowered Multi-Agent Systemscited by 7
  6. 2025Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignmentcited by 7
  7. 2026SWAY: A Counterfactual Computational Linguistic Approach to Measuring and Mitigating Sycophancycited by 6
  8. 2025A Hybrid Transformer Model for Fake News Detection: Leveraging Bayesian Optimization and Bidirectional Recurrent Unitcited by 3
  9. 2025On the Robustness of Verbal Confidence of LLMs in Adversarial Attackscited by 3
  10. 2026Seeing Beyond 8bits: Subjective and Objective Quality Assessment of HDR-UGC Videoscited by 1
  11. 2025Performance Gains of LLMs With Humans in a World of LLMs Versus Humanscited by 0
  12. 2026Vision-Language Model Confidence Is Not a Property of the Answercited by 0
1 more
  1. 2026Jailbreaks for Black-Box Uncertainty Quantification in Large Reasoning Modelscited by 0