mlsanityData to 14 Oct 2026

TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systems

arXiv 2311.11315 ↗First posted 19 Nov 2023cs.AIcited by 39

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated proficiency in addressing tasks that necessitate a combination of task planning and the usage of external tools that require a blend of task planning and the utilization of external tools, such as APIs. However, real-world complex systems present three prevalent challenges concerning task planning and tool usage: (1) The real system usually has a vast array of APIs, so it is impossible to feed the descriptions of all APIs to the prompt of LLMs as the token length is limited; (2) the real system is designed for handling complex tasks, and the base LLMs can hardly plan a correct sub-task order and API-calling order for such tasks; (3) Similar semantics and functionalities among APIs in real systems create challenges for both LLMs and even humans in distinguishing between them. In response, this paper introduces a comprehensive framework aimed at enhancing the Task Planning and Tool Usage (TPTU) abilities of LLM-based agents operating within real-world systems. Our framework comprises three key components designed to address these challenges: (1) the API Retriever selects the most pertinent APIs for the user task among the extensive array available; (2) LLM Finetuner tunes a base LLM so that the finetuned LLM can be more capable for task planning and API calling; (3) the Demo Selector adaptively retrieves different demonstrations related to hard-to-distinguish APIs, which is further used for in-context learning to boost the final performance. We validate our methods using a real-world commercial system as well as an open-sourced academic dataset, and the outcomes clearly showcase the efficacy of each individual component as well as the integrated framework.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 59 of its 63 references that are in the map. First posted on arXiv 19 Nov 2023.

Builds on

59 of 63 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMs19 Oct
  2. 2023FireAct: Toward Language Agent Fine-tuning9 Oct
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep
  4. 2023A Survey on Large Language Model based Autonomous Agents22 Aug
  5. 2022A Generalist Agent12 May

Other references · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  9. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  11. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  12. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
38 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  9. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  12. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  13. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  14. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  15. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  16. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  17. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  18. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  19. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  20. 2023Augmented Language Models: a Surveycited by 403
  21. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  22. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  23. 2023Tool Learning with Foundation Modelscited by 302
  24. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
  25. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 294
  26. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 289
  27. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 261
  28. 2017Efficient Natural Language Response Suggestion for Smart Replycited by 229
  29. 2022TALM: Tool Augmented Language Modelscited by 197
  30. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  31. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  32. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 156
  33. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  34. 2022Enabling Conversational Interaction with Mobile UI using Large Language Modelscited by 66
  35. 2023TableGPT: Towards Unifying Tables, Nature Language and Commands into One GPTcited by 59
  36. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  37. 2023SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Modelscited by 57
  38. 2023ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Applicationcited by 48

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 23; 2025: 11; 2026: 3.

39 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  2. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  3. 2024Tool Learning with Large Language Models: A Surveycited by 223
  4. 2024Continual Learning for Large Language Models: A Surveycited by 147
  5. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  6. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 96
  7. 2024ToolGen: Unified Tool Retrieval and Calling via Generationcited by 74
  8. 2024Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool Retrievalcited by 52
  9. 2024From LLM to Conversational Agent: A Memory Enhanced Architecture with Fine-Tuning of Large Language Modelscited by 43
  10. 2023Controlling Large Language Model-based Agents for Large-Scale Decision-Making: An Actor-Critic Approachcited by 38
  11. 2024Large Language Model-based Human-Agent Collaboration for Complex Task Solvingcited by 38
  12. 2024SQLBench: A Comprehensive Evaluation for Text-to-SQL Capabilities of Large Language Modelscited by 32
27 more
  1. 2024ToolPlanner: A Tool Augmented LLM for Multi Granularity Instructions with Path Planning and Feedbackcited by 29
  2. 2024PET-SQL: A Prompt-Enhanced Two-Round Refinement of Text-to-SQL with Cross-consistencycited by 27
  3. 2024Octopus: On-device language model for function calling of software APIscited by 27
  4. 2024DMQR-RAG: Diverse Multi-Query Rewriting for RAGcited by 27
  5. 2024Synergistic Multi-Agent Framework with Trajectory Learning for Knowledge-Intensive Taskscited by 23
  6. 2024A Trembling House of Cards? Mapping Adversarial Attacks against Language Agentscited by 22
  7. 2024MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language Modelscited by 17
  8. 2025Lifelong Safety Alignment for Language Modelscited by 15
  9. 2023Towards Responsible Generative AI: A Reference Architecture for Designing Foundation Model based Agentscited by 14
  10. 2024ChatHuman: Chatting about 3D Humans with Toolscited by 11
  11. 2025AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoningcited by 11
  12. 2024QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learningcited by 10
  13. 2024Probing the Safety Response Boundary of Large Language Models via Unsafe Decoding Path Generationcited by 10
  14. 2025Distilling On-device Language Models for Robot Planning with Minimal Human Interventioncited by 10
  15. 2024SoAy: A Solution-based LLM API-using Methodology for Academic Information Seekingcited by 7
  16. 2025Improving Retrospective Language Agents via Joint Policy Gradient Optimizationcited by 4
  17. 2026PORTS: Preference-Optimized Retrievers for Tool Selection with Large Language Modelscited by 4
  18. 2024Skin Cancer Detection utilizing Deep Learning: Classification of Skin Lesion Images using a Vision Transformercited by 3
  19. 2025LoSemB: Logic-Guided Semantic Bridging for Inductive Tool Retrievalcited by 3
  20. 2025AgenticIE: An Adaptive Agent for Information Extraction from Complex Regulatory Documentscited by 3
  21. 2024SEAL: Suite for Evaluating API-use of LLMscited by 2
  22. 2025NGENT: Next-Generation AI Agents Must Integrate Multi-Domain Abilities to Achieve Artificial General Intelligencecited by 2
  23. 2025AgentArcEval: An Architecture Evaluation Method for Foundation Model based Agentscited by 2
  24. 2026PEARL: Plan Exploration and Adaptive Reinforcement Learning for Multihop Tool Usecited by 1
  25. 2024Exploring Genre and Success Classification through Song Lyrics using DistilBERT: A Fun NLP Venturecited by 0
  26. 2025TUMS: Enhancing Tool-use Abilities of LLMs with Multi-structure Handlerscited by 0
  27. 2026Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage
  2. 2025TPS-Bench: Evaluating AI Agents' Tool Planning & Scheduling Abilities in Compounding Tasks
  3. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
  4. 2025Task Memory Engine (TME): Enhancing State Awareness for Multi-Step LLM Agent Tasks
  5. 2024A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learning
  6. 2024ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents
  7. 2024Creating an LLM-based AI-agent: A high-level methodology towards enhancing LLMs with APIs
  8. 2024Understanding the planning of LLM agents: A survey