mlsanityData to 7 Oct 2026

Automated Design of Agentic Systems

arXiv 2408.08435 ↗First posted 15 Aug 2024cs.AIcited by 360

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Researchers are investing substantial effort in developing powerful general-purpose agents, wherein Foundation Models are used as modules within agentic systems (e.g. Chain-of-Thought, Self-Reflection, Toolformer). However, the history of machine learning teaches us that hand-designed solutions are eventually replaced by learned solutions. We describe a newly forming research area, Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs, including inventing novel building blocks and/or combining them in new ways. We further demonstrate that there is an unexplored yet promising approach within ADAS where agents can be defined in code and new agents can be automatically discovered by a meta agent programming ever better ones in code. Given that programming languages are Turing Complete, this approach theoretically enables the learning of any possible agentic system: including novel prompts, tool use, workflows, and combinations thereof. We present a simple yet effective algorithm named Meta Agent Search to demonstrate this idea, where a meta agent iteratively programs interesting new agents based on an ever-growing archive of previous discoveries. Through extensive experiments across multiple domains including coding, science, and math, we show that our algorithm can progressively invent agents with novel designs that greatly outperform state-of-the-art hand-designed agents. Importantly, we consistently observe the surprising result that agents invented by Meta Agent Search maintain superior performance even when transferred across domains and models, demonstrating their robustness and generality. Provided we develop it safely, our work illustrates the potential of an exciting new research direction toward automatically designing ever-more powerful agentic systems to benefit humanity.

Key facts. Cited by 360 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 68 of its 120 references that are in the map. First posted on arXiv 15 Aug 2024.

Builds on

68 of 120 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Symbolic Learning Enables Self-Evolving Agents26 Jun
  2. 2024EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithms20 Jun
  3. 2024Scaling Large Language Model-based Multi-Agent Collaboration11 Jun
  4. 2024Offline Training of Language Model Agents with Functions as Learnable Weights17 Feb
  5. 2023A Survey on Large Language Model based Autonomous Agents22 Aug
  6. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors21 Aug
  7. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework1 Aug

Other references · 61

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  11. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  12. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
38 more
  1. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  5. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  8. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  9. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  10. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  11. 2023Large Language Models as Optimizerscited by 830
  12. 2022PAL: Program-aided Language Modelscited by 635
  13. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  14. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  15. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  16. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  17. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  18. 2015Illuminating search spaces by mapping elitescited by 406
  19. 2016Learning to reinforcement learncited by 369
  20. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  21. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  22. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  23. 2024Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Modelcited by 258
  24. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  25. 2024Tool Learning with Large Language Models: A Surveycited by 221
  26. 2023Take a Step Back: Evoking Reasoning via Abstraction in Large Language Modelscited by 157
  27. 2023ExpertPrompting: Instructing Large Language Models to be Distinguished Expertscited by 140
  28. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 128
  29. 2023Language Model Crossover: Variation through Few-Shot Promptingcited by 119
  30. 2017Quality and Diversity Optimization: A Unifying Modular Frameworkcited by 111
  31. 2024Self-Discover: Large Language Models Self-Compose Reasoning Structurescited by 108
  32. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 103
  33. 2019AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligencecited by 84
  34. 2024AutoFlow: Automated Workflow Generation for Large Language Model Agentscited by 76
  35. 2024OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Codecited by 67
  36. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 66
  37. 2024Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMscited by 62
  38. 2024Discovering Preference Optimization Algorithms with and for Large Language Modelscited by 38

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 94; 2026: 245.

360 papers.

Sort:

By area:

  1. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
  2. 2024AFlow: Automating Agentic Workflow Generationcited by 395
  3. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  4. 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 248
  5. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  6. 2025Multi-agent Architecture Search via Agentic Supernetcited by 164
  7. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  8. 2026Towards End-to-End Automation of AI Researchcited by 150
  9. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  10. 2024AgentSquare: Automatic LLM Agent Search in Modular Design Spacecited by 141
  11. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
  12. 2026MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agentscited by 129
38 more
  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnessescited by 119
  3. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systemscited by 112
  4. 2025Multi-Agent Design: Optimizing Agents with Better Prompts and Topologiescited by 106
  5. 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?cited by 105
  6. 2025Multi-Agent Collaboration via Evolving Orchestrationcited by 102
  7. 2024TableGPT2: A Large Multimodal Model with Tabular Data Integrationcited by 69
  8. 2024Archon: An Architecture Search Framework for Inference-Time Techniquescited by 59
  9. 2025SagaLLM: Context Management, Validation, and Transaction Guarantees for Multi-Agent LLM Planningcited by 58
  10. 2025A Self-Improving Coding Agentcited by 58
  11. 2025FlowReasoner: Reinforcing Query-Level Meta-Agentscited by 56
  12. 2025EvoFlow: Evolving Diverse Agentic Workflows On The Flycited by 49
  13. 2025MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systemscited by 45
  14. 2025ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoningcited by 41
  15. 2026HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundrycited by 37
  16. 2024LLM4AD: A Platform for Algorithm Design with Large Language Modelcited by 33
  17. 2025X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMscited by 33
  18. 2026Dive into Claude Code: The Design Space of Today's and Future AI Agent Systemscited by 33
  19. 2025Generative to Agentic AI: Survey, Conceptualization, and Challengescited by 32
  20. 2025Internet of Agents: Fundamentals, Applications, and Challengescited by 31
  21. 2024ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systemscited by 30
  22. 2025Adaptive Graph Pruning for Multi-Agent Communicationcited by 28
  23. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 27
  24. 2025MAS-ZERO: Designing Multi-Agent Systems with Zero Supervisioncited by 26
  25. 2026MAS-Orchestra: Understanding and Improving Multi-Agent Reasoning Through Holistic Orchestration and Controlled Benchmarkscited by 26
  26. 2026SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agentscited by 25
  27. 2026Stop Comparing LLM Agents Without Disclosing the Harnesscited by 25
  28. 2024Aviary: training language agents on challenging scientific taskscited by 23
  29. 2025Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systemscited by 22
  30. 2025DyFlow: Dynamic Workflow Framework for Agentic Reasoningcited by 22
  31. 2026ReCreate: Reasoning and Creating Domain Agents Driven by Experiencecited by 22
  32. 2026From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agentscited by 22
  33. 2026From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flawscited by 21
  34. 2025Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executorscited by 20
  35. 2025Agentic AI Needs a Systems Theorycited by 19
  36. 2025VFlow: Discovering Optimal Agentic Workflows for Verilog Generationcited by 19
  37. 2025RobustFlow: Towards Robust Agentic Workflow Generationcited by 19
  38. 2025Failure-Driven Workflow Refinementcited by 19

50 most cited of 360 All 360 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Inefficiencies of Meta Agents for Agent Design
  2. 2026AlgoEvo: Self-Evolving Agentic Search for Automated Algorithm Discovery
  3. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities
  4. 2026Agentic Design Patterns: A System-Theoretic Framework
  5. 2026Learning to Continually Learn via Meta-learning Agentic Memory Designs
  6. 2025MetaAgent: Automatically Constructing Multi-Agent Systems Based on Finite State Machines
  7. 2025SwarmAgentic: Towards Fully Automated Agentic System Generation via Swarm Intelligence
  8. 2025Control Plane as a Tool: A Scalable Design Pattern for Agentic AI Systems