mlsanityData to 7 Oct 2026

VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks

arXiv 2406.08394 ↗First posted 12 Jun 2024cs.CVcited by 136

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We present VisionLLM v2, an end-to-end generalist multimodal large model (MLLM) that unifies visual perception, understanding, and generation within a single framework. Unlike traditional MLLMs limited to text output, VisionLLM v2 significantly broadens its application scope. It excels not only in conventional visual question answering (VQA) but also in open-ended, cross-domain vision tasks such as object localization, pose estimation, and image generation and editing. To this end, we propose a new information transmission mechanism termed "super link", as a medium to connect MLLM with task-specific decoders. It not only allows flexible transmission of task information and gradient feedback between the MLLM and multiple downstream decoders but also effectively resolves training conflicts in multi-tasking scenarios. In addition, to support the diverse range of tasks, we carefully collected and combed training data from hundreds of public vision and vision-language tasks. In this way, our model can be joint-trained end-to-end on hundreds of vision language tasks and generalize to these tasks using a set of shared parameters through different user prompts, achieving performance comparable to task-specific models. We believe VisionLLM v2 will offer a new perspective on the generalization of MLLMs.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 181 of its 229 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

181 of 229 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  2. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception5 Mar
  3. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling19 Feb
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  5. 2023Generative Multimodal Models are In-Context Learners20 Dec
  6. 2023Osprey: Pixel Understanding with Visual Instruction Tuning15 Dec
  7. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts1 Dec
  8. 2023LLMGA: Multimodal Large Language Model based Generation Assistant27 Nov
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  10. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents9 Nov
  11. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  12. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Models4 Oct
8 more
  1. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  2. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  5. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  7. 2023Generating Images with Multimodal Language Models26 May
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 161

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  12. 2023Segment Anythingcited by 8,158
38 more
  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2016Layer Normalizationcited by 5,579
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  13. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  14. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  15. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  16. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  17. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  18. 2019Towards VQA Models That Can Readcited by 1,696
  19. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  20. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  21. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  22. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  23. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  24. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  25. 2021Grounded Language-Image Pre-trainingcited by 1,147
  26. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  27. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  28. 2016Modeling Context in Referring Expressionscited by 1,054
  29. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  30. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  31. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  32. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  33. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  34. 2016A Diagram Is Worth A Dozen Imagescited by 793
  35. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  36. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  37. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  38. 2017DOTA: A Large-scale Dataset for Object Detection in Aerial Imagescited by 622

50 most cited of 161 All 161 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 77; 2026: 33.

136 papers.

Sort:

By area:

  1. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  2. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  3. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  4. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  5. 2025TerraMind: Large-Scale Generative Multimodality for Earth Observationcited by 101
  6. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
  7. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 83
  8. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  9. 2025Detect Anything via Next Point Predictioncited by 66
  10. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 59
  11. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  12. 2025Uni-Sign: Toward Unified Sign Language Understanding at Scalecited by 55
38 more
  1. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 52
  2. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  3. 2024GeoGround: A Unified Large Vision-Language Model for Remote Sensing Visual Groundingcited by 46
  4. 2025VisRL: Intention-Driven Visual Perception via Reinforced Reasoningcited by 39
  5. 2024MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversitycited by 38
  6. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 37
  7. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  8. 2025A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1cited by 34
  9. 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understandingcited by 30
  10. 2024RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Expertscited by 29
  11. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  12. 2025Falcon: A Remote Sensing Vision-Language Foundation Model (Technical Report)cited by 28
  13. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
  14. 2024GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Groundingcited by 26
  15. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  16. 2025Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Modelscited by 22
  17. 2024MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understandingcited by 21
  18. 2025HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Modelcited by 19
  19. 2025AutoP2C: An LLM-Based Agent Framework for Code Repository Generation from Multimodal Content in Academic Paperscited by 18
  20. 2025Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinementcited by 18
  21. 2025PixelVLA: Advancing Pixel-level Understanding in Vision-Language-Action Modelcited by 16
  22. 2025UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understandingcited by 15
  23. 2024MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automationcited by 14
  24. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  25. 2024Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentationcited by 13
  26. 2025Region-based Cluster Discrimination for Visual Representation Learningcited by 13
  27. 2025NuGrounding: A Multi-View 3D Visual Grounding Framework in Autonomous Drivingcited by 12
  28. 2025Guiding Audio Editing with Audio Language Modelcited by 11
  29. 2024MM-CamObj: A Comprehensive Multimodal Dataset for Camouflaged Object Scenarioscited by 10
  30. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  31. 2024Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Modelscited by 10
  32. 2025Learning Streaming Video Representation via Multitask Trainingcited by 10
  33. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  34. 2025Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naive Integration via Multi-Objective Balanced Coveringcited by 10
  35. 2025LMM-Det: Make Large Multimodal Models Excel in Object Detectioncited by 10
  36. 2025V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMscited by 9
  37. 2024Multimodal 3D Reasoning Segmentation with Complex Scenescited by 8
  38. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8

50 most cited of 136 All 136 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023InfMLLM: A Unified Framework for Visual-Language Tasks
  2. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
  4. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  5. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones
  6. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
  7. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
  8. 2023IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models