# ModelScope-Agent: Building Your Customizable Agent System with Open-source Large Language Models

arXiv: https://arxiv.org/abs/2309.00986 · HTML: https://mlsanity.com/paper/2309.00986/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 2 Sep 2023.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023)
- [Tool Learning with Foundation Models](https://mlsanity.com/paper/2304.08354/) (2023)
- [GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction](https://mlsanity.com/paper/2305.18752/) (2023)
- [ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented Instruction Tuning for Digital Human](https://mlsanity.com/paper/2304.07849/) (2023)

## Built on by (most cited first, 24 of 24)

- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception](https://mlsanity.com/paper/2401.16158/) (2024)
- [Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives](https://mlsanity.com/paper/2312.11970/) (2023)
- [Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration](https://mlsanity.com/paper/2406.01014/) (2024)
- [AgentScope: A Flexible yet Robust Multi-Agent Platform](https://mlsanity.com/paper/2402.14034/) (2024)
- [GTA: A Benchmark for General Tool Agents](https://mlsanity.com/paper/2407.08713/) (2024)
- [CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databases](https://mlsanity.com/paper/2408.03910/) (2024)
- [Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applications](https://mlsanity.com/paper/2411.14110/) (2024)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices](https://mlsanity.com/paper/2407.03913/) (2024)
- [SAFEFLOW: A Principled Protocol for Trustworthy and Transactional Autonomous Agent Systems](https://mlsanity.com/paper/2506.07564/) (2025)
- [Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs](https://mlsanity.com/paper/2503.05139/) (2025)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [Very Large-Scale Multi-Agent Simulation in AgentScope](https://mlsanity.com/paper/2407.17789/) (2024)
- [Beyond ReAct: A Planner-Centric Framework for Complex Tool-Augmented LLM Reasoning](https://mlsanity.com/paper/2511.10037/) (2025)
- [Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Risk](https://mlsanity.com/paper/2502.15865/) (2025)
- [GraphTeam: Facilitating Large Language Model-based Graph Analysis via Multi-Agent Collaboration](https://mlsanity.com/paper/2410.18032/) (2024)
- [Technical Report of TeleChat2, TeleChat2.5 and T1](https://mlsanity.com/paper/2507.18013/) (2025)
- [Professional Agents -- Evolving Large Language Models into Autonomous Experts with Human-Level Competencies](https://mlsanity.com/paper/2402.03628/) (2024)
- [Long-Video Audio Synthesis with Multi-Agent Collaboration](https://mlsanity.com/paper/2503.10719/) (2025)
- [RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting](https://mlsanity.com/paper/2606.00147/) (2026)
- [MoRAgent: Parameter Efficient Agent Tuning with Mixture-of-Roles](https://mlsanity.com/paper/2512.21708/) (2025)
- [Invariant-based Robust Weights Watermark for Large Language Models](https://mlsanity.com/paper/2507.08288/) (2025)
- [MATRIX: Multimodal Agent Tuning for Robust Tool-Use Reasoning](https://mlsanity.com/paper/2510.08567/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have recently demonstrated remarkable capabilities to comprehend human intentions, engage in reasoning, and design planning-like behavior. To further unleash the power of LLMs to accomplish complex tasks, there is a growing trend to build agent framework that equips LLMs, such as ChatGPT, with tool-use abilities to connect with massive external APIs. In this work, we introduce ModelScope-Agent, a general and customizable agent framework for real-world applications, based on open-source LLMs as controllers. It provides a user-friendly system library, with customizable engine design to support model training on multiple open-source LLMs, while also enabling seamless integration with both model APIs and common APIs in a unified way. To equip the LLMs with tool-use abilities, a comprehensive framework has been proposed spanning over tool-use data collection, tool retrieval, tool registration, memory control, customized model training, and evaluation for practical real-world applications. Finally, we showcase ModelScopeGPT, a real-world intelligent assistant of ModelScope Community based on the ModelScope-Agent framework, which is able to connect open-source LLMs with more than 1000 public AI models and localized community knowledge in ModelScope. The ModelScope-Agent libraryfootnotehttps://github.com/modelscope/modelscope-agent and online demofootnotehttps://modelscope.cn/studios/damo/ModelScopeGPT/summary are now publicly available.