mlsanityData to 14 Oct 2026

LLM Agents Making Agent Tools

arXiv 2502.11705 ↗First posted 17 Feb 2025cs.CL cs.AI cs.LGcited by 48

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Tool use has turned large language models (LLMs) into powerful agents that can perform complex multi-step tasks by dynamically utilising external software components. However, these tools must be implemented in advance by human developers, hindering the applicability of LLM agents in domains demanding large numbers of highly specialised tools, like in life sciences and medicine. Motivated by the growing trend of scientific studies accompanied by public code repositories, we propose ToolMaker, an agentic framework that autonomously transforms papers with code into LLM-compatible tools. Given a GitHub URL and short task description, ToolMaker autonomously installs dependencies and generates code to perform the task, using a closed-loop self-correction mechanism for debugging. To evaluate our approach, we introduce a benchmark comprising 15 complex computational tasks spanning various domains with over 100 unit tests to assess correctness and robustness. Our method correctly implements 80% of the tasks, substantially outperforming current state-of-the-art software engineering agents. ToolMaker therefore is a step towards fully autonomous agent-based scientific workflows. Our code and benchmark are publicly available at https://github.com/KatherLab/ToolMaker.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 31 of its 61 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

31 of 61 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agents23 Jul
  2. 2024Agentless: Demystifying LLM-based Software Engineering Agents1 Jul
  3. 2023ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code16 Nov
  4. 2023ChatDev: Communicative Agents for Software Development16 Jul

Other references · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  7. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  8. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  9. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  10. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  11. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
  12. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
15 more
  1. 2023Large Language Models as Tool Makerscited by 303
  2. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 218
  3. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 161
  4. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 129
  5. 2024Empowering Biomedical Discovery with AI Agentscited by 118
  6. 2023On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)cited by 85
  7. 2024Phikon-v2, A large and public feature extractor for biomarker predictioncited by 74
  8. 2024FlowMap: High-Quality Camera Poses, Intrinsics, and Depth via Gradient Descentcited by 61
  9. 2024LegalAgentBench: Evaluating LLM Agents in Legal Domaincited by 59
  10. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositoriescited by 52
  11. 2024AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodologycited by 46
  12. 2024AgentMD: Empowering Language Agents for Risk Prediction with Large-Scale Clinical Tool Learningcited by 33
  13. 2024Automating Exploratory Proteomics Research via Language Modelscited by 11
  14. 2024Autonomous Artificial Intelligence Agents for Clinical Decision Making in Oncologycited by 6
  15. 2023What is the Impact of Releasing Code with Publications? Statistics from the Machine Learning, Robotics, and Control Communitiescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 21; 2026: 27.

48 papers.

Sort:

By area:

  1. 2025Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directionscited by 265
  2. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 167
  3. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 71
  4. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  5. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 42
  6. 2025AgentOrchestra: Orchestrating Multi-Agent Intelligence with the Tool-Environment-Agent(TEA) Protocolcited by 39
  7. 2025CASCADE: Cumulative Agentic Skill Creation through Autonomous Development and Evolutioncited by 34
  8. 2025Generative to Agentic AI: Survey, Conceptualization, and Challengescited by 33
  9. 2025RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoningcited by 20
  10. 2026Uncovering Security Threats and Architecting Defenses in Autonomous Agents: A Case Study of OpenClawcited by 20
  11. 2025From Glue-Code to Protocols: A Critical Analysis of A2A and MCP Integration for Scalable Agent Systemscited by 14
  12. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 14
36 more
  1. 2025Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physicscited by 11
  2. 2025Code2MCP: Transforming Code Repositories into MCP Servicescited by 10
  3. 2026What Makes a Good LLM Agent for Real-world Penetration Testing?cited by 10
  4. 2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomycited by 9
  5. 2026Uncertainty-Aware Clarification in LLM Agents with Information Gaincited by 8
  6. 2025LLM-Agent-Controller: A Universal Multi-Agent Large Language Model System as a Control Engineercited by 5
  7. 2025Open Agent Specification (Agent Spec): A Unified Representation for AI Agentscited by 5
  8. 2025Agentic Systems in Radiology: Design, Applications, Evaluation, and Challengescited by 5
  9. 2026Deep Tabular Research via Continual Experience-Driven Executioncited by 5
  10. 2026RADAR: Redundancy-Aware Diffusion for Multi-Agent Communication Structure Generationcited by 5
  11. 2025Proof-of-Use: Mitigating Tool-Call Hacking in Deep Research Agentscited by 4
  12. 2025Unifying Dynamic Tool Creation and Cross-Task Experience Sharing through Cognitive Memory Architecturecited by 4
  13. 2026Experience-Driven Multi-Agent Systems Are Training-free Context-aware Earth Observerscited by 4
  14. 2026Learning to Rewrite Tool Descriptions for Reliable LLM-Agent Tool Usecited by 4
  15. 2026FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Taskscited by 4
  16. 2025Giving AI Agents Access to Cryptocurrency and Smart Contracts Creates New Vectors of AI Harmcited by 3
  17. 2026LAMMI-Pathology: A Tool-Centric Bottom-Up LVLM-Agent Framework for Molecularly Informed Medical Intelligence in Pathologycited by 3
  18. 2025EH-Benchmark Ophthalmic Hallucination Benchmark and Agent-Driven Top-Down Traceable Reasoning Workflowcited by 2
  19. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  20. 2026Project Synapse: A Hierarchical Multi-Agent Framework with Hybrid Memory for Autonomous Resolution of Last-Mile Delivery Disruptionscited by 1
  21. 2026SAGE: Tool-Augmented LLM Task Solving Strategies in Scalable Multi-Agent Environmentscited by 1
  22. 2026Beyond Task Completion: A Verification-vs.-Conformance Gap in Tool-Evolving Agentscited by 1
  23. 2026Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearningcited by 1
  24. 2026Do Self-Evolving Skills Generalize to Held-Out Tasks?cited by 1
  25. 2025Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verificationcited by 0
  26. 2026Safe to Check, Unsafe to Use: Relinking at the Compression Boundary of LLM Agentscited by 0
  27. 2026A Multi-cluster Boundary Learning Method for Out-of-Scope Intent Detection via MiniLM Embeddingcited by 0
  28. 2026Eta Given Delta: Defining LLM Tool Efficiency With Marginal Tool Utilitycited by 0
  29. 2026Scalable LLM Agent Tool Access in the Cloudcited by 0
  30. 2026VideoResearcher: Self-Improving Tool Design for Long-Video Understandingcited by 0
  31. 2026MCPGen: Benchmarking LLMs on Executable MCPWorkflow Developmentcited by 0
  32. 2026Prompt Injection Detection for Email Agents Through Attack Chain Modelingcited by 0
  33. 2026Improving Medical Calculation of LLMs with Embedded Codingcited by 0
  34. 2026Recursive Improvement of a Differentiable Scientific Software Ecosystemcited by 0
  35. 2026SciCurateBench: Curating Experimental Data from Heterogeneous Scientific Literature with Verbal-Action Agentscited by 0
  36. 2026Coupled Harness--Model Evolution for Scientific Equation Discoverycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AutoAgent: A Fully-Automated and Zero-Code Framework for LLM Agents
  2. 2025ToolFuzz -- Automated Agent Tool Testing
  3. 2025A Survey on Code Generation with LLM-based Agents
  4. 2024AutoFlow: Automated Workflow Generation for Large Language Model Agents
  5. 2026ToolRosella: Translating Code Repositories into Standardized Tools for Scientific Agents
  6. 2026Evaluating LLM Agents on Automated Software Analysis Tasks
  7. 2026Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report
  8. 2026DOCSCHISEL: Adaptive Tool Documentation Optimization Framework for LLM Agents