mlsanityData to 14 Oct 2026

InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language

arXiv 2305.05662 ↗First posted 9 May 2023cs.CVcited by 96

Not on the map. Why?

Cited by 38 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present an interactive visual framework named InternGPT, or iGPT for short. The framework integrates chatbots that have planning and reasoning capabilities, such as ChatGPT, with non-verbal instructions like pointing movements that enable users to directly manipulate images or videos on the screen. Pointing (including gestures, cursors, etc.) movements can provide more flexibility and precision in performing vision-centric tasks that require fine-grained control, editing, and generation of visual content. The name InternGPT stands for interaction, nonverbal, and chatbots. Different from existing interactive systems that rely on pure language, by incorporating pointing instructions, the proposed iGPT significantly improves the efficiency of communication between users and chatbots, as well as the accuracy of chatbots in vision-centric tasks, especially in complicated visual scenarios where the number of objects is greater than 2. Additionally, in iGPT, an auxiliary control mechanism is used to improve the control capability of LLM, and a large vision-language model termed Husky is fine-tuned for high-quality multi-modal dialogue (impressing ChatGPT-3.5-turbo with 93.89% GPT-4 Quality). We hope this work can spark new ideas and directions for future interactive visual systems. Welcome to watch the code at https://github.com/OpenGVLab/InternGPT.

Key facts. Cited by 96 papers in the map (data through 14 Oct 2026). Builds on 70 of its 83 references that are in the map. First posted on arXiv 9 May 2023.

Builds on

70 of 83 references.

References · 70

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2022Training language models to follow instructions with human feedbackcited by 15,971
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  10. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  11. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  12. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
38 more
  1. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  2. 2023Segment Anythingcited by 8,234
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  11. 2014Going Deeper with Convolutionscited by 4,241
  12. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  13. 2022A ConvNet for the 2020scited by 3,579
  14. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  15. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  16. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  17. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  18. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  19. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  20. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  21. 2018SlowFast Networks for Video Recognitioncited by 1,867
  22. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  23. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  24. 2021ViViT: A Video Vision Transformercited by 1,620
  25. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  26. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  27. 2021Grounded Language-Image Pre-trainingcited by 1,156
  28. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  29. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  30. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  31. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  32. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  33. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  34. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  35. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  36. 2022PAL: Program-aided Language Modelscited by 652
  37. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  38. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 36; 2024: 37; 2025: 19; 2026: 4.

96 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  4. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  5. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  6. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  7. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  8. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  9. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  10. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  11. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
  12. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
38 more
  1. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  2. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  3. 2024Tool Learning with Large Language Models: A Surveycited by 223
  4. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 222
  5. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  6. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 139
  7. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  8. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  9. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 133
  10. 2023SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimationcited by 117
  11. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 116
  12. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  13. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 116
  14. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  15. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 96
  16. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  17. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  18. 2023ChatPose: Chatting about 3D Human Posecited by 72
  19. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 71
  20. 2023A Comprehensive Survey on Segment Anything Model for Vision and Beyondcited by 64
  21. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  22. 2023ControlLLM: Augment Language Models with Tools by Searching on Graphscited by 59
  23. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 59
  24. 2024Needle In A Multimodal Haystackcited by 58
  25. 2025GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensingcited by 57
  26. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usagecited by 55
  27. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 53
  28. 2023Going Denser with Open-Vocabulary Part Segmentationcited by 51
  29. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Modelscited by 51
  30. 2024VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videoscited by 51
  31. 2023A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineeringcited by 45
  32. 2023CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Updatecited by 43
  33. 2024MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversitycited by 38
  34. 2024Towards Vision-Language Geo-Foundation Model: A Surveycited by 36
  35. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  36. 2025Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spacescited by 33
  37. 2023Tokenize Anything via Promptingcited by 31
  38. 2023ViT-Lens: Towards Omni-modal Representationscited by 29

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023ChatGPT for Robotics: Design Principles and Model Abilities
  2. 2024VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework
  3. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Models
  4. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models
  5. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policy
  6. 2023InterAct: Exploring the Potentials of ChatGPT as a Cooperative Agent
  7. 2026ClickAIXR: On-Device Multimodal Vision-Language Interaction with Real-World Objects in Extended Reality
  8. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn