mlsanityData to 14 Oct 2026

TAI3: Testing Agent Integrity in Interpreting User Intent

arXiv 2506.07524 ↗First posted 9 Jun 2025cs.SE cs.AI cs.CYcited by 9

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

LLM agents are increasingly deployed to automate real-world tasks by invoking APIs through natural language instructions. While powerful, they often suffer from misinterpretation of user intent, leading to the agent's actions that diverge from the user's intended goal, especially as external toolkits evolve. Traditional software testing assumes structured inputs and thus falls short in handling the ambiguity of natural language. We introduce TAI3, an API-centric stress testing framework that systematically uncovers intent integrity violations in LLM agents. Unlike prior work focused on fixed benchmarks or adversarial inputs, TAI3 generates realistic tasks based on toolkits' documentation and applies targeted mutations to expose subtle agent errors while preserving user intent. To guide testing, we propose semantic partitioning, which organizes natural language tasks into meaningful categories based on toolkit API parameters and their equivalence classes. Within each partition, seed tasks are mutated and ranked by a lightweight predictor that estimates the likelihood of triggering agent errors. To enhance efficiency, TAI3 maintains a datatype-aware strategy memory that retrieves and adapts effective mutation patterns from past cases. Experiments on 80 toolkit APIs demonstrate that TAI3 effectively uncovers intent integrity violations, significantly outperforming baselines in both error-exposing rate and query efficiency. Moreover, TAI3 generalizes well to stronger target models using smaller LLMs for test generation, and adapts to evolving APIs across domains.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 81 of its 115 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

81 of 115 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputs27 Apr

Other references · 80

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  6. 2022Red Teaming Language Models with Language Modelscited by 1,026
  7. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  8. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  9. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  11. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  12. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
38 more
  1. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  2. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
  3. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  4. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  5. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 538
  6. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  7. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  8. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 491
  9. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
  10. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 429
  11. 2018Generating Natural Language Adversarial Examplescited by 424
  12. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  13. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  14. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 381
  15. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
  16. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
  17. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 306
  18. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  19. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  20. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  21. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 216
  22. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  23. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  24. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 178
  25. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 171
  26. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 158
  27. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  28. 2024Long-form factuality in large language modelscited by 155
  29. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
  30. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 145
  31. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 133
  32. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
  33. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoningcited by 124
  34. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 119
  35. 2023An LLM can Fool Itself: A Prompt-Based Adversarial Attackcited by 116
  36. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agentscited by 115
  37. 2024LLM Agents can Autonomously Hack Websitescited by 109
  38. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 82

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attackscited by 10
  2. 2026When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agentscited by 8
  3. 2026Semia: Auditing Agent Skills via Constraint-Guided Representation Synthesiscited by 7
  4. 2025Temporal Logic-Based Multi-Vehicle Backdoor Attacks against Offline RL Agents in End-to-end Autonomous Drivingcited by 4
  5. 2026Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safetycited by 2
  6. 2026Evaluating Implicit Regulatory Compliance in LLM Tool Invocation via Logic-Guided Synthesiscited by 1
  7. 2026MemoPhishAgent: Memory-Augmented Multi-Modal LLM Agent for Phishing URL Detectioncited by 1
  8. 2026Natural-Language Workflows Are Not Software Yet: Artifact-Driven Compilation for Reliable Agent Executioncited by 0
  9. 2026Do Tool Calls Execute as Intended? Measuring and Repairing Intent-Execution Correspondence in LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ToolFuzz -- Automated Agent Tool Testing
  2. 2026Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications
  3. 2025ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases
  4. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  5. 2026ARGUS: Defending LLM Agents Against Context-Aware Prompt Injection
  6. 2026BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click
  7. 2025SIEVE: Selective Integrity Verification and Escalation for Defending LLM Agents against Indirect Prompt Injection
  8. 2026MANTRA: Synthesizing SMT-Validated Compliance Benchmarks for Tool-Using LLM Agents