mlsanityData to 14 Oct 2026

TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs

arXiv 2303.16434 ↗First posted 29 Mar 2023cs.AI cs.CLcited by 184

Not on the map. Why?

Cited by 69 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Artificial Intelligence (AI) has made incredible progress recently. On the one hand, advanced foundation models like ChatGPT can offer powerful conversation, in-context learning and code generation abilities on a broad range of open-domain tasks. They can also generate high-level solution outlines for domain-specific tasks based on the common sense knowledge they have acquired. However, they still face difficulties with some specialized tasks because they lack enough domain-specific data during pre-training or they often have errors in their neural network computations on those tasks that need accurate executions. On the other hand, there are also many existing models and systems (symbolic-based or neural-based) that can do some domain-specific tasks very well. However, due to the different implementation or working mechanisms, they are not easily accessible or compatible with foundation models. Therefore, there is a clear and pressing need for a mechanism that can leverage foundation models to propose task solution outlines and then automatically match some of the sub-tasks in the outlines to the off-the-shelf models and systems with special functionalities to complete them. Inspired by this, we introduce TaskMatrix.AI as a new AI ecosystem that connects foundation models with millions of APIs for task completion. Unlike most previous work that aimed to improve a single AI model, TaskMatrix.AI focuses more on using existing foundation models (as a brain-like central system) and APIs of other AI models and systems (as sub-task solvers) to achieve diversified tasks in both digital and physical domains. As a position paper, we will present our vision of how to build such an ecosystem, explain each key component, and use study cases to illustrate both the feasibility of this vision and the main challenges we need to address next.

Key facts. Cited by 184 papers in the map (data through 14 Oct 2026). Builds on 27 of its 30 references that are in the map. First posted on arXiv 29 Mar 2023.

Builds on

27 of 30 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  11. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
15 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  3. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  4. 2022Galactica: A Large Language Model for Sciencecited by 744
  5. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  6. 2022PAL: Program-aided Language Modelscited by 652
  7. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  8. 2023Augmented Language Models: a Surveycited by 403
  9. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
  10. 2022Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofscited by 291
  11. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 197
  12. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  13. 2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
  14. 2021Screen Parsing: Towards Reverse Engineering of UI Models from Screenshotscited by 32
  15. 2022Language modeling via stochastic processescited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 79; 2024: 51; 2025: 30; 2026: 24.

184 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  3. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  4. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  5. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  6. 2023A Comprehensive Overview of Large Language Modelscited by 648
  7. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  8. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  9. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  10. 2023Large Language Models as Tool Makerscited by 303
  11. 2023Tool Learning with Foundation Modelscited by 302
  12. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
38 more
  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  2. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  3. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 255
  4. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 251
  5. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  6. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 240
  7. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 225
  8. 2024Tool Learning with Large Language Models: A Surveycited by 223
  9. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 217
  10. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 184
  11. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
  12. 2023An LLM Compiler for Parallel Function Callingcited by 179
  13. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 169
  14. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 158
  15. 2024Continual Learning for Large Language Models: A Surveycited by 147
  16. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 144
  17. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  18. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  19. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  20. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 132
  21. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 129
  22. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  23. 2023A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directionscited by 105
  24. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  25. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
  26. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 96
  27. 2023CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Modelscited by 93
  28. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 90
  29. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  30. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 79
  31. 2023Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflowcited by 76
  32. 2024Unified Hallucination Detection for Multimodal Large Language Modelscited by 75
  33. 2025DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agentscited by 66
  34. 2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65
  35. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuningcited by 65
  36. 2024Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmarkcited by 63
  37. 2024LLM With Tools: A Surveycited by 61
  38. 2023Grammar Prompting for Domain-Specific Language Generation with Large Language Modelscited by 58

50 most cited of 184 All 184 (JSON)