mlsanityData to 7 Oct 2026

VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks

arXiv 2305.11175 ↗First posted 18 May 2023cs.CVcited by 499

Not on the map. Why?

Cited by 215 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have notably accelerated progress towards artificial general intelligence (AGI), with their impressive zero-shot capacity for user-tailored tasks, endowing them with immense potential across a range of applications. However, in the field of computer vision, despite the availability of numerous powerful vision foundation models (VFMs), they are still restricted to tasks in a pre-defined form, struggling to match the open-ended task capabilities of LLMs. In this work, we present an LLM-based framework for vision-centric tasks, termed VisionLLM. This framework provides a unified perspective for vision and language tasks by treating images as a foreign language and aligning vision-centric tasks with language tasks that can be flexibly defined and managed using language instructions. An LLM-based decoder can then make appropriate predictions based on these instructions for open-ended tasks. Extensive experiments show that the proposed VisionLLM can achieve different levels of task customization through language instructions, from fine-grained object-level to coarse-grained task-level customization, all with good results. It's noteworthy that, with a generalist LLM-based framework, our model can achieve over 60% mAP on COCO, on par with detection-specific models. We hope this model can set a new baseline for generalist vision and language models. The demo shall be released based on https://github.com/OpenGVLab/InternGPT. The code shall be released at https://github.com/OpenGVLab/VisionLLM.

Key facts. Cited by 499 papers in the map (data through 7 Oct 2026). Builds on 68 of its 81 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

68 of 81 references.

References · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2022Training language models to follow instructions with human feedbackcited by 15,712
  9. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  10. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  11. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  12. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
38 more
  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2020End-to-End Object Detection with Transformerscited by 7,384
  3. 2017Mask R-CNNcited by 7,259
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  13. 2015VQA: Visual Question Answeringcited by 2,489
  14. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  15. 2022Visual Prompt Tuningcited by 1,674
  16. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  17. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  18. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  19. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  20. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  21. 2016Modeling Context in Referring Expressionscited by 1,054
  22. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  23. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  24. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  25. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  26. 2022A Generalist Agentcited by 802
  27. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  28. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  29. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  30. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  31. 2016Self-critical Sequence Training for Image Captioningcited by 544
  32. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  33. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  34. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  35. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  36. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  37. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  38. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 99; 2024: 229; 2025: 137; 2026: 34.

499 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  6. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  7. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  8. 2023A Survey on Multimodal Large Language Modelscited by 898
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  10. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  11. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  12. 2023A Comprehensive Overview of Large Language Modelscited by 639
38 more
  1. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  2. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  3. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  4. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  5. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  6. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  7. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  8. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  9. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  10. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  11. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  12. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  13. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  14. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  15. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  16. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  17. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  18. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  19. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  20. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  21. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  22. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  23. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  24. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  25. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  26. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  27. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  28. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  29. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  30. 2024LLaGA: Large Language and Graph Assistantcited by 177
  31. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  32. 2023AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Modelscited by 169
  33. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  34. 2023ChatDB: Augmenting LLMs with Databases as Their Symbolic Memorycited by 136
  35. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  36. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  37. 2023On the Challenges and Perspectives of Foundation Models for Medical Image Analysiscited by 135
  38. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134

50 most cited of 499 All 499 (JSON)