mlsanityData to 14 Oct 2026

AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvement

arXiv 2502.00757 ↗First posted 2 Feb 2025cs.CR cs.AI cs.NEcited by 14

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

also in 1 more
Abstract (arXiv)

Scaffolding Large Language Models (LLMs) into multi-agent systems often improves performance on complex tasks, but the safety impact of such scaffolds has not been thoroughly explored. We introduce AgentBreeder, a framework for multi-objective self-improving evolutionary search over scaffolds. We evaluate discovered scaffolds on widely recognized reasoning, mathematics, and safety benchmarks and compare them with popular baselines. In "blue" mode, we see a 79.4% average uplift in safety benchmark performance while maintaining or improving capability scores. In "red" mode, we find adversarially weak scaffolds emerging concurrently with capability optimization. Our work demonstrates the risks of multi-agent scaffolding and provides a framework for mitigating them. Code is available at https://github.com/jrosseruk/AgentBreeder.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 42 of its 63 references that are in the map. First posted on arXiv 2 Feb 2025.

Builds on

42 of 63 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks7 Nov
  2. 2024G"odel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement6 Oct
  3. 2024Agent-Oriented Planning in Multi-Agent Systems3 Oct
  4. 2024ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systems2 Sep
  5. 2024Automated Design of Agentic Systems15 Aug
  6. 2024EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithms20 Jun
  7. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors21 Aug
  8. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate14 Aug

Other references · 34

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2023A Survey of Large Language Modelscited by 2,576
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  11. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  12. 2022Red Teaming Language Models with Language Modelscited by 1,026
22 more
  1. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  2. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  3. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  4. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  5. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  6. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  7. 2015Illuminating search spaces by mapping elitescited by 425
  8. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  9. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  10. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
  11. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 178
  12. 2023ExpertPrompting: Instructing Large Language Models to be Distinguished Expertscited by 142
  13. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  14. 2024SciAgents: Automating scientific discovery through multi-agent intelligent graph reasoningcited by 94
  15. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safetycited by 80
  16. 2024Self-Evolving Multi-Agent Collaboration Networks for Software Developmentcited by 74
  17. 2023Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPTcited by 71
  18. 2025Trading Inference-Time Compute for Adversarial Robustnesscited by 61
  19. 2024MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmarkcited by 41
  20. 2023DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacycited by 34
  21. 2024Polaris: A Safety-focused LLM Constellation Architecture for Healthcarecited by 28
  22. 2024A Superalignment Framework in Autonomous Driving with Large Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 7.

14 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 75
  3. 2025Demonstrations of Integrity Attacks in Multi-Agent Systemscited by 8
  4. 2025SyncLoop: A Multimodal Dual-Loop Framework for Self-Improving Mathematical Reasoningcited by 8
  5. 2026Information-Theoretic Privacy Control for Sequential Multi-Agent LLM Systemscited by 8
  6. 2026Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systemscited by 6
  7. 2025Opponent Shaping in LLM Agentscited by 2
  8. 2025Reliable agent engineering should integrate machine-compatible organizational principlescited by 2
  9. 2026Deep Reasoning in General Purpose Agents via Structured Meta-Cognitioncited by 2
  10. 2026Mitigating loss of control in advanced AI systems through instrumental goal trajectoriescited by 1
  11. 2025An Aristotelian ontology of instrumental goals: Structural features to be managed and not failures to be eliminatedcited by 0
  12. 2026A three-dimensional typology of agency for advanced AI systemscited by 0
2 more
  1. 2026MOAE: Multi-Objective Agent Evolution with Pareto-Preserving Searchcited by 0
  2. 2026AgentSWE: Can Coding Agents Build the Agent You Actually Want?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Models
  2. 2026Evo-Bench: Can Language Models Improve Agent Harness?
  3. 2026AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety
  4. 2026SHE: Trajectory-driven Safety Harness Evolution for LLM Agents
  5. 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
  6. 2026HERA: Harness-Environment Co-Evolution for Reliable Agentic Abstention
  7. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  8. 2026An Empirical Study of Multi-Agent Collaboration for Automated Research