Agentic Adversarial Rewriting Exposes Architectural Vulnerabilities in Black-Box NLP Pipelines

arXiv 2604.23483 ↗First posted 26 Apr 2026cs.AIcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Multi-component natural language processing (NLP) pipelines are increasingly deployed for high-stakes decisions, yet no existing adversarial method can test their robustness under realistic conditions: binary-only feedback, no gradient access, and strict query budgets. We formalize this strict black-box threat model and propose a two-agent evasion framework operating in a semantic perturbation space. An Attacker Agent generates meaning-preserving rewrites while a Prompt Optimization Agent refines the attack strategy using only binary decision feedback within a 10-query budget. Evaluated against four evidence-based misinformation detection pipelines, the framework achieves evasion rates of 19.95 to 40.34% on modern large language model (LLM) based systems, compared to at most 3.90% for token-level perturbation baselines that rely on surrogate models because they cannot operate under our threat model. A legacy system relying on static lexical retrieval exhibits near-total vulnerability 97.02%, establishing a lower bound that exposes how architectural choices govern the attack surface. Evasion effectiveness is associated with three architectural properties: evidence retrieval mechanism, retrieval-inference coupling, and baseline classification accuracy. The iterative prompt optimization yields the largest marginal gains against the most robust targets, confirming that adaptive strategy discovery is essential when evasion is non-trivial. Analysis of successful rewrites reveals four exploitation patterns, each targeting failures at distinct pipeline stages. A pattern-informed defense reduces the evasion rate by up to 65.18%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 43 of its 66 references that are in the map. First posted on arXiv 26 Apr 2026.

Builds on

43 of 66 references.

References · 43

2017Proximal Policy Optimization Algorithmscited by 15,215
2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
2019The Curious Case of Neural Text Degenerationcited by 2,916
2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
2024Qwen2 Technical Reportcited by 2,120
2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443

31 more

2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
2022Red Teaming Language Models with Language Modelscited by 1,026
2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 491
2018Generating Natural Language Adversarial Examplescited by 424
2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 381
2017"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detectioncited by 363
2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 345
2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 267
2023AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Webcited by 144
2020Contextualized Perturbation for Textual Adversarial Attackcited by 136
2023An LLM can Fool Itself: A Prompt-Based Adversarial Attackcited by 116
2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 88
2020SAFER: A Structure-free Approach for Certified Robustness to Adversarial Word Substitutionscited by 73
2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Modelscited by 67
2021Certified Robustness to Text Adversarial Attacks by Randomized [MASK]cited by 57
2020Generating Natural Language Attacks in a Hard Label Black Box Settingcited by 50
2024Phantom: General Backdoor Attacks on Retrieval Augmented Language Generationcited by 49
2023Towards Reliable Misinformation Mitigation: Generalization, Uncertainty, and GPT-4cited by 45
2022Synthetic Disinformation Attacks on Automated Fact Verification Systemscited by 44
2022SemAttack: Natural Textual Attacks via Different Semantic Spacescited by 42
2024Evidence-backed Fact Checking using RAG and Few-Shot In-Context Learning with LLMscited by 24
2021A Strong Baseline for Query Efficient Attacks in a Black Box Settingcited by 16
2022Text Adversarial Purification as Defense against Adversarial Attackscited by 16
2023TextGuard: Provable Defense against Backdoor Attacks on Text Classificationcited by 16
2024HQA-Attack: Toward High Quality Black-Box Hard-Label Adversarial Attack on Textcited by 16
2022Fact-Saboteurs: A Taxonomy of Evidence Manipulation Attacks against Fact-Verification Systemscited by 12
2024Attacking Misinformation Detection Using Adversarial Examples Generated by Language Modelscited by 10
2023LimeAttack: Local Explainable Method for Textual Hard-Label Adversarial Attackcited by 9
2023Don't Retrain, Just Rewrite: Countering Adversarial Perturbations by Rewriting Textcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way