mlsanityData to 7 Oct 2026

MLLM training and visual instruction tuning

1,468 papers3 this weekLow confidence in this grouping: the papers may belong to more than one topic, or the group may changeLow confidence in this grouping: the papers may belong to more than one topic, or the group may change · About

Keywords: multimodal, mllms, language, large, visual

Key facts. MLLM training and visual instruction tuning: 1,468 papers on arXiv since 2021 (data through 7 Oct 2026). 336 of its 1,468 mapped papers (23%) cite “Improved Baselines with Visual Instruction Tuning” (2023, arXiv 2310.03744). 272 of its 1,468 mapped papers (19%) cite “InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning” (2023, arXiv 2305.06500). 22 new papers 17 Sep–7 Oct.

What’s new

17 Sep – 7 Oct 2026

22 new papers, 17 Sep – 7 Oct; 3 this week.

  1. This week · 3
  2. From Pixel to Coding: Evaluating the Figure Reproduction Capabilities of MLLMs also fits: VLM benchmarks and visual reasoning
  3. Multimodal LLMs Can Learn to Read Brain Signals: A Vision--Language Model for Unified Multi-Task EEG Decoding also fits: Multimodal emotion understanding
  4. Two Vectors Replace In-Context Demos: Structured Task Adaptation via Embeddings also fits: Prompt learning and adaptation of CLIP
  5. Earlier · 19
  6. A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models also fits: VLM benchmarks and visual reasoning
  7. Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs
  8. Architecture-Dependent Fusion Pathways in MLLMs
  9. Revealing Epistemic Uncertainty in MLLMs via Causal-Invariant Masking
  10. Slow-Fast Multi-Teacher On-Policy Distillation for Capability Preservation
  11. Gestalt: Large Multimodal Interplay Model also fits: Image editing and quality assessment
  12. MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models also fits: VLM benchmarks and visual reasoning
All 22
  1. MM-FinEval: A Multi-Task Multimodal Benchmark for Real-World Financial Forecasting also fits: Chart understanding and anomaly detection
  2. ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuning also fits: Embodied and 3D vision-language models
  3. Calibrate the Decisions That Change the Future: On-Policy Post-Training Quantization for Multimodal Large Language Models also fits: Embodied and 3D vision-language models
  4. How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining also fits: Visual token pruning and efficiency
  5. MoR-MLLM: Mixture of Recursions for Efficient Multimodal Large Language Models also fits: Visual token pruning and efficiency
  6. FORUM: Frozen Outputs Reconciled Using Model Agreement for Visual Grounding
  7. BIRD: Distilling Decision Boundaries into Rationales for MLLM Adaptation also fits: VLM benchmarks and visual reasoning
  8. The Alignment Illusion in Multimodal Large Language Models
  9. Virtual Encoders in Multimodal Transformers
  10. BAS-OPD: Budget-Aware Selective On-Policy Self-Distillation for Fine-Grained Multimodal Perception also fits: Image editing and quality assessment
  11. Pretraining of Medical Visual Encoders Toward Multi-modal Large Language Models also fits: Medical vision-language models
  12. Offline Multimodal Large Language Models for Decision Support in Air Operations also fits: Document understanding and multimodal RAG

Foundations

Most cited by papers here

  1. Improved Baselines with Visual Instruction Tuning2023cited by 336 here
  2. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning2023cited by 272 here
  3. MMBench: Is Your Multi-modal Model an All-around Player?2023cited by 270 here
  4. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models2023cited by 201 here
  5. MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities2023cited by 169 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 115 here
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 221 here
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 272 here
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 194 here
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 201 here
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 130 here
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 169 here
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 116 here
  9. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 93 here
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 160 here
  11. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 70 here
13 more it builds on, not in this chain
  1. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 86 here
  2. 2023A Survey on Multimodal Large Language Modelscited by 89 here
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 80 here
  4. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 102 here
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 270 here
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 164 here
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 149 here
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 336 here
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 148 here
  10. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 75 here
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 79 here
  12. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 84 here
  13. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 112 here

The 24 most cited of 1,978.

Browse

30 of 1,468, ranked 6–35

  1. 6Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution2024160
  2. 7InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks2023148
  3. 8SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension2023130
  4. 9ShareGPT4V: Improving Large Multi-Modal Models with Better Captions2023116
  5. 10LLaVA-OneVision: Easy Visual Task Transfer2024112
  6. 11Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic2023102
  7. 12A Survey on Multimodal Large Language Models202389
  8. 13Otter: A Multi-Modal Model with In-Context Instruction Tuning202386
  9. 14Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs202484
  10. 15How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites202479
All 30
  1. 16Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs202475
  2. 17Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling202470
  3. 18mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration202368
  4. 19VILA: On Pre-training for Visual Language Models202360
  5. 20OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents202359
  6. 21Aligning Large Multimodal Models with Factually Augmented RLHF202358
  7. 22MiniCPM-V: A GPT-4V Level MLLM on Your Phone202455
  8. 23Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models202352
  9. 24The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)202348
  10. 25BLINK: Multimodal Large Language Models Can See but Not Perceive202446
  11. 26MoE-LLaVA: Mixture of Experts for Large Vision-Language Models202442
  12. 27Generative Multimodal Models are In-Context Learners202341
  13. 28SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models202339
  14. 29DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding202438
  15. 30NExT-GPT: Any-to-Any Multimodal LLM202338
  16. 31LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models202337
  17. 32MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training202436
  18. 33Honeybee: Locality-enhanced Projector for Multimodal LLM202335
  19. 34Qwen3-VL Technical Report202532
  20. 35MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning202331