mlsanityData to 14 Oct 2026

SafeSearch: Automated Red-Teaming of LLM-Based Search Agents

arXiv 2509.23694 ↗First posted 28 Sep 2025cs.AI cs.CL cs.CRcited by 4

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Search agents connect LLMs to the Internet, enabling them to access broader and more up-to-date information. However, this also introduces a new threat surface: unreliable search results can mislead agents into producing unsafe outputs. Real-world incidents and our two in-the-wild observations show that such failures can occur in practice. To study this threat systematically, we propose SafeSearch, an automated red-teaming framework that is scalable, cost-efficient, and lightweight, enabling sandboxed safety evaluation of search agents. Using this, we generate 300 test cases spanning five risk categories (e.g., misinformation and prompt injection) and evaluate three search agent scaffolds across 17 representative LLMs. Our results reveal substantial vulnerabilities in LLM-based search agents, with the highest ASR reaching 90.5% for GPT-4.1-mini in a search-workflow setting. Moreover, we find that common defenses, such as reminder prompting, offer limited protection. Overall, SafeSearch provides a practical way to measure and improve the safety of LLM-based search agents.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 65 of its 75 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

65 of 75 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systems16 Jun
  2. 2025Defeating Prompt Injections by Design24 Mar
  3. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents19 Dec
  4. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents17 Dec
  5. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
  6. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
  7. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage17 Sep
  8. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 57

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2025Gemma 3 Technical Reportcited by 1,824
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  12. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
38 more
  1. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  3. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  4. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  5. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  6. 2024Self-Rewarding Language Modelscited by 681
  7. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  8. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  9. 2025Kimi K2: Open Agentic Intelligencecited by 567
  10. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  11. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  12. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 433
  13. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  14. 2024A StrongREJECT for Empty Jailbreakscited by 385
  15. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  16. 2024Measuring short-form factuality in large language modelscited by 373
  17. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  18. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
  19. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  20. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  21. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 267
  22. 2025Agentic Retrieval-Augmented Generation: A Survey on Agentic RAGcited by 249
  23. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 249
  24. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 217
  25. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 208
  26. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 200
  27. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
  28. 2024Long-form factuality in large language modelscited by 155
  29. 2023On the Risk of Misinformation Pollution with Large Language Modelscited by 128
  30. 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidencecited by 113
  31. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 102
  32. 2025WideSearch: Benchmarking Agentic Broad Info-Seekingcited by 69
  33. 2025A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challengescited by 62
  34. 2025Trading Inference-Time Compute for Adversarial Robustnesscited by 61
  35. 2024"Glue pizza and eat rocks" -- Exploiting Vulnerabilities in Retrieval-Augmented Generative Modelscited by 60
  36. 2025SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledgecited by 54
  37. 2025A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applicationscited by 50
  38. 2024Adversarial Search Engine Optimization for Large Language Modelscited by 40

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Your Agentic LLMs Secretly Encode Indirect Prompt-Injection Exposure in Hidden Statescited by 7
  2. 2025AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learningcited by 6
  3. 2026Is Deep Research Reliable? Misleading Knowledge Induces False Conclusionscited by 4
  4. 2026EcoGEO: Trajectory-Aware Evidence Ecosystems for Web-Enabled LLM Search Agentscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents
  2. 2026SearchAttack: Red-Teaming LLMs against Knowledge-to-Action Threats under Online Web Search
  3. 2026Relevance as a Vulnerability: How Web Retrieval Degrades Safety Alignment in LLM Agents
  4. 2025When Search Goes Wrong: Red-Teaming Web-Augmented Large Language Models
  5. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases
  6. 2026How Much Can We Trust LLM Search Agents? Measuring Endorsement Vulnerability to Web Content Manipulation
  7. 2025SafeArena: Evaluating the Safety of Autonomous Web Agents
  8. 2025Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design