mlsanityData to 7 Oct 2026

Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models

arXiv 2303.04671 ↗First posted 8 Mar 2023cs.CVcited by 659

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

ChatGPT is attracting a cross-field interest as it provides a language interface with remarkable conversational competency and reasoning capabilities across many domains. However, since ChatGPT is trained with languages, it is currently not capable of processing or generating images from the visual world. At the same time, Visual Foundation Models, such as Visual Transformers or Stable Diffusion, although showing great visual understanding and generation capabilities, they are only experts on specific tasks with one-round fixed inputs and outputs. To this end, We build a system called Visual ChatGPT, incorporating different Visual Foundation Models, to enable the user to interact with ChatGPT by 1) sending and receiving not only languages but also images 2) providing complex visual questions or visual editing instructions that require the collaboration of multiple AI models with multi-steps. 3) providing feedback and asking for corrected results. We design a series of prompts to inject the visual model information into ChatGPT, considering models of multiple inputs/outputs and models that require visual feedback. Experiments show that Visual ChatGPT opens the door to investigating the visual roles of ChatGPT with the help of Visual Foundation Models. Our system is publicly available at urlhttps://github.com/microsoft/visual-chatgpt.

Key facts. Cited by 659 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 52 of its 62 references that are in the map. First posted on arXiv 8 Mar 2023.

Builds on

52 of 62 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep
  3. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language1 Apr

Other references · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
37 more
  1. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2021Vision Transformers for Dense Predictioncited by 1,789
  11. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  13. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  14. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  15. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  16. 2021Scaling Vision Transformerscited by 993
  17. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  18. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  19. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  20. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  21. 2015Holistically-Nested Edge Detectioncited by 590
  22. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  23. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  24. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  25. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  26. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  27. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  28. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  29. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  30. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  31. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  32. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  33. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  34. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  35. 2016Semantic Compositional Networks for Visual Captioningcited by 50
  36. 2021Towards Light-weight and Real-time Line Segment Detectioncited by 33
  37. 2020Text as Neural Operator: Image Manipulation by Text Instructioncited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 256; 2024: 196; 2025: 118; 2026: 89.

659 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  10. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  11. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
38 more
  1. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  2. 2023A Survey on Multimodal Large Language Modelscited by 898
  3. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  4. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  5. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  6. 2023A Comprehensive Overview of Large Language Modelscited by 639
  7. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  8. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  9. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  10. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  11. 2023Segment Everything Everywhere All at Oncecited by 546
  12. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  13. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  14. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  15. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  16. 2024Understanding the planning of LLM agents: A surveycited by 414
  17. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  18. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  19. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  20. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  21. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  22. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  23. 2023Tool Learning with Foundation Modelscited by 302
  24. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  25. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  26. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  27. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  28. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  29. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  30. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  31. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  32. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  33. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  34. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  35. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  36. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  37. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  38. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264

50 most cited of 659 All 659 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting with Diffusion Transformers
  2. 2024GPTDrawer: Enhancing Visual Synthesis through ChatGPT
  3. 2026Chatting with Images for Introspective Visual Thinking
  4. 2023ChatGPT is not all you need. A State of the Art Review of large Generative AI models
  5. 2023DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention
  6. 2025InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions
  7. 2023Multimodal Analysis Of Google Bard And GPT-Vision: Experiments In Visual Reasoning
  8. 2025VisualQuest: A Benchmark for Abstract Visual Reasoning in MLLMs