mlsanityData to 7 Oct 2026

LLM Agents Factory: Retrieval of Domain-Specific LLM Agents

arXiv 2608.09934 ↗First posted 20 May 2026cs.CL cs.AIcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large language model (LLM) agents improve task performance by decomposing problems into role-specialized behaviors. However, their practical deployment is often limited by the computational cost and instability associated with the on-the-fly agent design for each user request. To address this, we present LLM Agents Factory, a retrieval-based framework that constructs domain-specific and Wikipedia-grounded agents on demand using a base of over 20K predetermined agent profiles. Our framework supports two modes: (1) agent profile retrieval via semantic search and (2) distillation into a compact model fine-tuned for direct agent generation. Experiments on MMLU, BIG-bench, and BIG-bench Hard in a single-agent scenario demonstrate that our retrieval-based agent construction surpasses non-agent baselines in accuracy while matching AutoGen generation quality with a 120B backbone at a substantially lower inference cost. Our work reveals that retrieval from a structured agent repository provides a cost-efficient, accurate, and controllable alternative to dynamic agent generation, responding to the strict demands of industrial applications. We provide the implementation code and the agent base in https://huggingface.co/frontier-ai/llm-agent-factory.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 28 of its 42 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

28 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023A Survey on Large Language Model based Autonomous Agents22 Aug
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society31 Mar

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2017Billion-scale similarity search with GPUscited by 2,477
  11. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
14 more
  1. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  5. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  6. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  7. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  8. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  9. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  10. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  11. 2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Usecited by 67
  12. 2019Uncovering the Semantics of Wikipedia Categoriescited by 10
  13. 2024GRILLBot In Practice: Lessons and Tradeoffs Deploying Large Language Models for Adaptable Conversational Task Assistantscited by 6
  14. 2026Ontology-to-tools compilation for executable semantic constraint enforcement in LLM agentscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Generalizability of Large Language Model-Based Agents: A Comprehensive Survey
  2. 2024AgentSquare: Automatic LLM Agent Search in Modular Design Space
  3. 2025AgentSwift: Efficient LLM Agent Design via Value-guided Hierarchical Search
  4. 2023Enhancing Open-Domain Task-Solving Capability of LLMs via Autonomous Tool Integration from GitHub
  5. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  6. 2026Toward Formalizing LLM-Based Agent Designs through Structural Context Modeling and Semantic Dynamics Analysis
  7. 2026Improving LLM-based Autonomous Web Agents with Filtering
  8. 2026Agent Gym: A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback