mlsanityData to 7 Oct 2026

ControlLLM: Augment Language Models with Tools by Searching on Graphs

arXiv 2310.17796 ↗First posted 26 Oct 2023cs.CV cs.MMcited by 59

Not on the map. Why?

Cited by 24 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present ControlLLM, a novel framework that enables large language models (LLMs) to utilize multi-modal tools for solving complex real-world tasks. Despite the remarkable performance of LLMs, they still struggle with tool invocation due to ambiguous user prompts, inaccurate tool selection and parameterization, and inefficient tool scheduling. To overcome these challenges, our framework comprises three key components: (1) a textittask decomposer that breaks down a complex task into clear subtasks with well-defined inputs and outputs; (2) a textitThoughts-on-Graph (ToG) paradigm that searches the optimal solution path on a pre-built tool graph, which specifies the parameter and dependency relations among different tools; and (3) an textitexecution engine with a rich toolbox that interprets the solution path and runs the tools efficiently on different computational devices. We evaluate our framework on diverse tasks involving image, audio, and video processing, demonstrating its superior accuracy, efficiency, and versatility compared to existing methods. The code is at https://github.com/OpenGVLab/ControlLLM.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 44 of its 55 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

44 of 55 references.

References · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
32 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  5. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  6. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  10. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  11. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  13. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  14. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  15. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  16. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  17. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  18. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  19. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  20. 2022Large Language Models Are Reasoning Teacherscited by 318
  21. 2023Tool Learning with Foundation Modelscited by 302
  22. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  23. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  24. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  25. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  26. 2023Generating Images with Multimodal Language Modelscited by 259
  27. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  28. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  29. 2022TALM: Tool Augmented Language Modelscited by 197
  30. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
  31. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  32. 2023CephGPT-4: An Interactive Multimodal Cephalometric Measurement and Diagnostic System with Visual Large Language Modelcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 25; 2025: 15; 2026: 13.

59 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  3. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  4. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  5. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  6. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  7. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  8. 2024Tool Learning with Large Language Models: A Surveycited by 221
  9. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  10. 2024Large Multimodal Agents: A Surveycited by 99
  11. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  12. 2024Demystifying Chains, Trees, and Graphs of Thoughtscited by 63
38 more
  1. 2024Needle In A Multimodal Haystackcited by 56
  2. 2026Graph-of-Skills: Dependency-Aware Structural Retrieval for Massive Agent Skillscited by 53
  3. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  4. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  5. 2023CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Updatecited by 42
  6. 2024Can Graph Learning Improve Planning in LLM-based Agents?cited by 42
  7. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  8. 2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Modelscited by 34
  9. 2026GraSP: Graph-Structured Skill Compositions for LLM Agentscited by 25
  10. 2025Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunitiescited by 23
  11. 2025Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environmentscited by 19
  12. 2024ALPINE: Unveiling the Planning Capability of Autoregressive Learning in Language Modelscited by 18
  13. 2025Graph RAG-Tool Fusioncited by 15
  14. 2025Tool-to-Agent Retrieval: Bridging Tools and Agents for Scalable LLM Multi-Agent Systemscited by 15
  15. 2024CATP-LLM: Empowering Large Language Models for Cost-Aware Tool Planningcited by 14
  16. 2025Graph-Augmented Large Language Model Agents: Current Progress and Future Prospectscited by 14
  17. 2024StrucText-Eval: Evaluating Large Language Model's Reasoning Ability in Structure-Rich Textcited by 13
  18. 2026Group of Skills: Group-Structured Skill Retrieval for Agent Skill Librariescited by 9
  19. 2024From Summary to Action: Enhancing Large Language Models for Complex Tasks with Open World APIscited by 8
  20. 2024SoAy: A Solution-based LLM API-using Methodology for Academic Information Seekingcited by 7
  21. 2024Tulip Agent -- Enabling LLM-Based Agents to Solve Tasks Using Large Tool Librariescited by 7
  22. 2024EMMA: Efficient Visual Alignment in Multi-Modal LLMscited by 7
  23. 2025Advancing and Benchmarking Personalized Tool Invocation for LLMscited by 7
  24. 2025Experience-Evolving Multi-Turn Tool-Use Agent with Hybrid Episodic-Procedural Memorycited by 6
  25. 2026PhGPO: Pheromone-Guided Policy Optimization for Long-Horizon Tool Planningcited by 6
  26. 2024NPHardEval4V: Dynamic Evaluation of Large Vision-Language Models with Effects of Visioncited by 5
  27. 2024From An LLM Swarm To A PDDL-Empowered HIVE: Planning Self-Executed Instructions In A Multi-Modal Junglecited by 4
  28. 2025Agent-as-a-Graph: Knowledge Graph-Based Tool and Agent Retrieval for LLM Multi-Agent Systemscited by 4
  29. 2023How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Modelcited by 3
  30. 2024Human-Centered LLM-Agent User Interface: A Position Papercited by 3
  31. 2025NaviAgent: Graph-Driven Bilevel Planning for Scalable Tool Orchestrationcited by 3
  32. 2026Graph-Based Self-Healing Tool Routing for Cost-Efficient LLM Agentscited by 3
  33. 2026IoT-Brain: Grounding LLMs for Semantic-Spatial Sensor Schedulingcited by 2
  34. 2026IEA: Amateur-Friendly Conversational Image Editing Agent via Three Stages of Multitask Alignmentcited by 2
  35. 2024RE-GAINS & EnChAnT: Intelligent Tool Manipulation Systems For Enhanced Query Responsescited by 1
  36. 2024Action Engine: Automatic Workflow Generation in FaaScited by 1
  37. 2025In-N-Out: A Parameter-Level API Graph Dataset for Tool Agentscited by 1
  38. 2026AudioX-Turbo: A Unified Framework for Efficient Anything-to-Audio Generationcited by 1

50 most cited of 59 All 59 (JSON)