mlsanityData to 7 Oct 2026

From Pixels to Prompts: Vision-Language Models

arXiv 2605.07544 ↗First posted 8 May 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

When you read a paper about a new Vision-Language Model today, it can be easy to forget how strange this idea would have sounded not so long ago. Teaching machines to see was already hard. Teaching them to read and generate language was already hard. Asking them to do both at once - and then to reason, answer questions, follow instructions, and sometimes even surprise us - still carries a quiet trace of science fiction, even as it becomes routine. This book was born from a simple feeling: it is too easy to get lost. The field moves quickly, new model names appear constantly, and the gap between "I know the buzzwords" and "I actually understand how this works" can feel uncomfortably wide. I have felt that gap many times. If you are holding this book, you probably have too. My goal is not to provide an exhaustive catalog of every dataset, benchmark, and new model variant. Instead, I want to offer something more modest - and, I hope, more durable: a clear mental map of Vision-Language Models. Enough structure that you can read new papers with confidence; enough intuition that you can design your own systems without feeling as if you are assembling LEGO bricks blindly.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 9 of its 90 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

9 of 90 references.

References · 9

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2025Qwen3-VL Technical Reportcited by 2,534
  7. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  8. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  9. 2025Ovis2.5 Technical Reportcited by 103

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Dreaming Out Loud: A Self-Synthesis Approach For Training Vision-Language Models With Developmentally Plausible Data
  2. 2025ReCo: Reminder Composition Mitigates Hallucinations in Vision-Language Models
  3. 2025NanoVLMs: How small can we go and still make coherent Vision Language Models?
  4. 2025Vision Language Models: A Survey of 26K Papers
  5. 2025Relative Drawing Identification Complexity is Invariant to Modality in Vision-Language Models
  6. 2026World Models in Words: Auditing Physical State-Transition Commitments in Vision-Language Models
  7. 2026Flatness Preserves Instruction Following in Vision-Language-Action Models
  8. 2026Vision Is Not Overhead: One-Pass Block Drafting for Lossless Speculative Decoding in Vision-Language Models