mlsanityData to 7 Oct 2026

Confidence Elicitation: A New Attack Vector for Large Language Models

arXiv 2502.04643 ↗First posted 7 Feb 2025cs.LG cs.CL cs.CRcited by 4

Not on the map. Why?

Abstract (arXiv)

A fundamental issue in deep learning has been adversarial robustness. As these systems have scaled, such issues have persisted. Currently, large language models (LLMs) with billions of parameters suffer from adversarial attacks just like their earlier, smaller counterparts. However, the threat models have changed. Previously, having gray-box access, where input embeddings or output logits/probabilities were visible to the user, might have been reasonable. However, with the introduction of closed-source models, no information about the model is available apart from the generated output. This means that current black-box attacks can only utilize the final prediction to detect if an attack is successful. In this work, we investigate and demonstrate the potential of attack guidance, akin to using output probabilities, while having only black-box access in a classification setting. This is achieved through the ability to elicit confidence from the model. We empirically show that the elicited confidence is calibrated and not hallucinated for current LLMs. By minimizing the elicited confidence, we can therefore increase the likelihood of misclassification. Our new proposed paradigm demonstrates promising state-of-the-art results on three datasets across two models (LLaMA-3-8B-Instruct and Mistral-7B-Instruct-V0.3) when comparing our technique to existing hard-label black-box attack methods that introduce word-level substitutions.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 46 of its 64 references that are in the map. First posted on arXiv 7 Feb 2025.

Builds on

46 of 64 references.

References · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2013Intriguing properties of neural networkscited by 5,114
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2016Adversarial examples in the physical worldcited by 1,762
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  10. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  11. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  12. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
34 more
  1. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  2. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  3. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  4. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  5. 2018Universal Sentence Encodercited by 485
  6. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  7. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  8. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  9. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  10. 2023Are aligned neural networks adversarially aligned?cited by 305
  11. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 282
  12. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  13. 2016Counter-fitting Word Vectors to Linguistic Constraintscited by 138
  14. 2023An LLM can Fool Itself: A Prompt-Based Adversarial Attackcited by 111
  15. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  16. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  17. 2019Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systemscited by 99
  18. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  19. 2024Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approachcited by 73
  20. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  21. 2021Towards Robustness Against Natural Language Word Substitutionscited by 68
  22. 2020It's Morphin' Time! Combating Linguistic Discrimination with Inflectional Perturbationscited by 64
  23. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 60
  24. 2020Generating Natural Language Attacks in a Hard Label Black Box Settingcited by 50
  25. 2020CAT-Gen: Improving Robustness in NLP Models via Controlled Adversarial Text Generationcited by 43
  26. 2023Second-Order Uncertainty Quantification: A Distance-Based Approachcited by 35
  27. 2024Reasoning Robustness of LLMs to Adversarial Typographical Errorscited by 31
  28. 2024Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Imagecited by 26
  29. 2024Finetuning Language Models to Emit Linguistic Expressions of Uncertaintycited by 25
  30. 2022TextHacker: Learning based Hybrid Local Search Algorithm for Text Hard-label Adversarial Attackcited by 10
  31. 2024Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers?cited by 10
  32. 2024Enhancing Adversarial Robustness of Vision-Language Models through Low-Rank Adaptationcited by 8
  33. 2024Extreme Miscalibration and the Illusion of Adversarial Robustnesscited by 4
  34. 2024SemRoDe: Macro Adversarial Training to Learn Representations That are Robust to Word-Level Attackscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Text Adversarial Attacks with Dynamic Outputscited by 2
  2. 2025Style Attack Disguise: When Fonts Become a Camouflage for Adversarial Intentcited by 2
  3. 2026TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language Modelscited by 1
  4. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0