mlsanityData to 7 Oct 2026

AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn

arXiv 2306.08640 ↗First posted 14 Jun 2023cs.CVcited by 97

Not on the map. Why?

Cited by 40 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent research on Large Language Models (LLMs) has led to remarkable advancements in general NLP AI assistants. Some studies have further explored the use of LLMs for planning and invoking models or APIs to address more general multi-modal user queries. Despite this progress, complex visual-based tasks still remain challenging due to the diverse nature of visual tasks. This diversity is reflected in two aspects: 1) Reasoning paths. For many real-life applications, it is hard to accurately decompose a query simply by examining the query itself. Planning based on the specific visual content and the results of each step is usually required. 2) Flexible inputs and intermediate results. Input forms could be flexible for in-the-wild cases, and involves not only a single image or video but a mixture of videos and images, e.g., a user-view image with some reference videos. Besides, a complex reasoning process will also generate diverse multimodal intermediate results, e.g., video narrations, segmented video clips, etc. To address such general cases, we propose a multi-modal AI assistant, AssistGPT, with an interleaved code and language reasoning approach called Plan, Execute, Inspect, and Learn (PEIL) to integrate LLMs with various tools. Specifically, the Planner is capable of using natural language to plan which tool in Executor should do next based on the current reasoning progress. Inspector is an efficient memory manager to assist the Planner to feed proper visual information into a specific tool. Finally, since the entire reasoning process is complex and flexible, a Learner is designed to enable the model to autonomously explore and discover the optimal solution. We conducted experiments on A-OKVQA and NExT-QA benchmarks, achieving state-of-the-art results. Moreover, showcases demonstrate the ability of our system to handle questions far more complex than those found in the benchmarks.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Builds on 67 of its 73 references that are in the map. First posted on arXiv 14 Jun 2023.

Builds on

67 of 73 references.

References · 67

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2023Segment Anythingcited by 8,158
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
38 more
  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  11. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  13. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  14. 2021Grounded Language-Image Pre-trainingcited by 1,147
  15. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  16. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  17. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  18. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  19. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  20. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  21. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  22. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  23. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  24. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  25. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  26. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  27. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  28. 2015Neural Module Networkscited by 375
  29. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  30. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  31. 2022Egocentric Video-Language Pretrainingcited by 231
  32. 2023OpenAGI: When LLM Meets Domain Expertscited by 196
  33. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  34. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  35. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  36. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  37. 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
  38. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 41; 2025: 26; 2026: 9.

97 papers.

Sort:

By area:

  1. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  4. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  5. 2023A Comprehensive Overview of Large Language Modelscited by 639
  6. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  7. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  8. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  9. 2024Tool Learning with Large Language Models: A Surveycited by 221
  10. 2023Video Understanding with Large Language Models: A Surveycited by 203
  11. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 115
  12. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
38 more
  1. 2024Large Multimodal Agents: A Surveycited by 99
  2. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  3. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  4. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  5. 2023Embodied Task Planning with Large Language Modelscited by 84
  6. 2024TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Taskscited by 70
  7. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
  8. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactionscited by 67
  9. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  10. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  11. 2024LLM With Tools: A Surveycited by 60
  12. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 56
  13. 2024What Are Tools Anyway? A Survey from the Language Model Perspectivecited by 53
  14. 2023LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordinationcited by 48
  15. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  16. 2023CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Updatecited by 42
  17. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  18. 2024m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Taskscited by 38
  19. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 36
  20. 2024Online Adaptation of Language Models with a Memory of Amortized Contextscited by 35
  21. 2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
  22. 2025LLM Pretraining with Continuous Conceptscited by 33
  23. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  24. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learningcited by 30
  25. 2023LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generationcited by 30
  26. 2024VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMscited by 29
  27. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 28
  28. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  29. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  30. 2025MDCrow: Automating Molecular Dynamics Workflows with Large Language Modelscited by 22
  31. 2024SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledgecited by 20
  32. 2024ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videoscited by 18
  33. 2024KoMA: Knowledge-driven Multi-agent Framework for Autonomous Driving with Large Language Modelscited by 16
  34. 2024Learning Video Context as Interleaved Multimodal Sequencescited by 16
  35. 2024EAGLE: Egocentric AGgregated Language-video Enginecited by 16
  36. 2024Adaptive Video Understanding Agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoningcited by 16
  37. 2024Towards Rationality in Language and Multimodal Agents: A Surveycited by 15
  38. 2024Autonomous Agents for Collaborative Task under Information Asymmetrycited by 15

50 most cited of 97 All 97 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework
  2. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models
  3. 2025An Efficient Training Pipeline for Reasoning Graphical User Interface Agents
  4. 2024What Is Missing in Multilingual Visual Reasoning and How to Fix It
  5. 2024POEM: Interactive Prompt Optimization for Enhancing Multimodal Reasoning of Large Language Models
  6. 2026Are We There Yet? Exploring the Capabilities of MLLMs in Assistive AI Applications
  7. 2026VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding
  8. 2025SeeingEye: Agentic Information Flow Unlocks Multimodal Reasoning In Text-only LLMs