mlsanityData to 14 Oct 2026

Counterfactual Evaluation for Blind Attack Detection in LLM-based Evaluation Systems

arXiv 2507.23453 ↗First posted 31 Jul 2025cs.CR cs.CLcited by 3

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

This paper investigates defenses for LLM-based evaluation systems against prompt injection. We formalize a class of threats called blind attacks, where a candidate answer is crafted independently of the true answer to deceive the evaluator. To counter such attacks, we propose a framework that augments Standard Evaluation (SE) with Counterfactual Evaluation (CFE), which re-evaluates the submission against a deliberately false ground-truth answer. An attack is detected if the system validates an answer under both standard and counterfactual conditions. Experiments show that while standard evaluation is highly vulnerable, our SE+CFE framework significantly improves security by boosting attack detection with minimal performance trade-offs.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 36 of its 46 references that are in the map. First posted on arXiv 31 Jul 2025.

Builds on

36 of 46 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Prompt Injection Detection and Mitigation via AI Multi-Agent NLP Frameworks14 Mar
  2. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judge26 Mar
  3. 2024Defending Against Indirect Prompt Injection Attacks With Spotlighting20 Mar
  4. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models21 Dec
  5. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 30

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  12. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
18 more
  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  2. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  3. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  4. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  5. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  6. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 286
  7. 2023Certifying LLM Safety against Adversarial Promptingcited by 266
  8. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 216
  9. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  10. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  11. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 144
  12. 2023Backdoor Attacks for In-Context Learning with Language Modelscited by 96
  13. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 74
  14. 2024TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarizationcited by 66
  15. 2024LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoningcited by 59
  16. 2023The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Riskscited by 36
  17. 2023LoFT: Local Proxy Fine-tuning For Improving Transferability Of Adversarial Attacks Against Large Language Modelcited by 21
  18. 2022TASA: Deceiving Question Answering Models by Twin Answer Sentences Attackcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysiscited by 30
  2. 2026Unsupervised Style Representation Learning for AI-Text Detection via Paraphrase Inversioncited by 0
  3. 2026EDCT-Bench: Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questions
  2. 2026Cordyceps: Covert Control Attacks on LLMs via Data Poisoning
  3. 2025From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMs
  4. 2025How Not to Detect Prompt Injections with an LLM
  5. 2026Did the LLM Answer the Question? A Semantic Reconstruction Game for Jailbreak Evaluation
  6. 2025Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitors
  7. 2025Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections
  8. 2025Enhancing Security in LLM Applications: A Performance Evaluation of Early Detection Systems