mlsanityData to 14 Oct 2026

Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questions

arXiv 2508.13214 ↗First posted 16 Aug 2025cs.CR cs.AIcited by 4

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs) have recently demonstrated strong emergent abilities in complex reasoning and zero-shot generalization, showing unprecedented potential for LLM-as-a-judge applications in education, peer review, and data quality evaluation. However, their robustness under prompt injection attacks, where malicious instructions are embedded into the content to manipulate outputs, remains a significant concern. In this work, we explore a frustratingly simple yet effective attack setting to test whether LLMs can be easily misled. Specifically, we evaluate LLMs on basic arithmetic questions (e.g., "What is 3 + 2?") presented as either multiple-choice or true-false judgment problems within PDF files, where hidden prompts are injected into the file. Our results reveal that LLMs are indeed vulnerable to such hidden prompt injection attacks, even in these trivial scenarios, highlighting serious robustness risks for LLM-as-a-judge applications.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 99 of its 111 references that are in the map. First posted on arXiv 16 Aug 2025.

Builds on

99 of 111 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judge26 Mar
  2. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts26 Feb
  3. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models21 Dec
  4. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection17 Aug

Other references · 95

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  10. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  11. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  12. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 766
38 more
  1. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  2. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  3. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  4. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  5. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  6. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  7. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 391
  8. 2016Solving General Arithmetic Word Problemscited by 360
  9. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  10. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 302
  11. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  12. 2023Mathematical Capabilities of ChatGPTcited by 268
  13. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 260
  14. 2023TextDiffuser: Diffusion Models as Text Painterscited by 229
  15. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 228
  16. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 200
  17. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 193
  18. 2024The Illusion of State in State-Space Modelscited by 187
  19. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  20. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 156
  21. 2022UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expressioncited by 137
  22. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 134
  23. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
  24. 2023An LLM can Fool Itself: A Prompt-Based Adversarial Attackcited by 116
  25. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 113
  26. 2024MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmarkcited by 105
  27. 2024MARG: Multi-Agent Review Generation for Scientific Paperscited by 104
  28. 2023Fast Attention Requires Bounded Entriescited by 102
  29. 2024AgentReview: Exploring Peer Review Dynamics with LLM Agentscited by 93
  30. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 79
  31. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Modelscited by 77
  32. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  33. 2023Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learningcited by 72
  34. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Modelscited by 68
  35. 2023Adversarial Attacks and Defenses in Large Language Models: Old and New Threatscited by 49
  36. 2024Make It Count: Text-to-Image Generation with an Accurate Number of Objectscited by 47
  37. 2023How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computationcited by 44
  38. 2023PromptBench: A Unified Library for Evaluation of Large Language Modelscited by 44

50 most cited of 95 All 95 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysiscited by 30
  2. 2025Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Countingcited by 18
  3. 2025Distractor Injection Attacks on Large Reasoning Models: Characterization and Defensecited by 2
  4. 2025Fundamental Limits of Crystalline Equivariant Graph Neural Networks: A Circuit Complexity Perspectivecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections
  2. 2025Multilingual Hidden Prompt Injection Attacks on LLM-Based Academic Reviewing
  3. 2024PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMs
  4. 2023Goal-Oriented Prompt Attack and Safety Evaluation for LLMs
  5. 2024Automatic and Universal Prompt Injection Attacks against Large Language Models
  6. 2025Investigating the Vulnerability of LLM-as-a-Judge Architectures to Prompt-Injection Attacks
  7. 2025Prompt Injection as an Emerging Threat: Evaluating the Resilience of Large Language Models
  8. 2025Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction