mlsanityData to 14 Oct 2026

Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models

arXiv 2312.06109 ↗First posted 11 Dec 2023cs.CVcited by 84

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Modern Large Vision-Language Models (LVLMs) enjoy the same vision vocabulary -- CLIP, which can cover most common vision tasks. However, for some special vision task that needs dense and fine-grained vision perception, e.g., document-level OCR or chart understanding, especially in non-English scenarios, the CLIP-style vocabulary may encounter low efficiency in tokenizing the vision knowledge and even suffer out-of-vocabulary problem. Accordingly, we propose Vary, an efficient and effective method to scale up the vision vocabulary of LVLMs. The procedures of Vary are naturally divided into two folds: the generation and integration of a new vision vocabulary. In the first phase, we devise a vocabulary network along with a tiny decoder-only transformer to produce the desired vocabulary via autoregression. In the next, we scale up the vanilla vision vocabulary by merging the new one with the original one (CLIP), enabling the LVLMs can quickly garner new features. Compared to the popular BLIP-2, MiniGPT4, and LLaVA, Vary can maintain its vanilla capabilities while enjoying more excellent fine-grained perception and understanding ability. Specifically, Vary is competent in new document parsing features (OCR or markdown conversion) while achieving 78.2% ANLS in DocVQA and 36.2% in MMVet. Our code will be publicly available on the homepage.

Key facts. Cited by 84 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 52 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

43 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Segment Anythingcited by 8,234
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  12. 2017Deep reinforcement learning from human preferencescited by 3,912
29 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  4. 2022Emergent Abilities of Large Language Modelscited by 2,512
  5. 2019Towards VQA Models That Can Readcited by 1,718
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  8. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  10. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  11. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  12. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  13. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  14. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  15. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  16. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  17. 2021OCR-free Document Understanding Transformercited by 407
  18. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  19. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  20. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 248
  21. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 245
  22. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 236
  23. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  24. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
  25. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 133
  26. 2022End-to-end Document Recognition and Understanding with Dessurtcited by 73
  27. 2021LaTr: Layout-Aware Transformer for Scene-Text VQAcited by 72
  28. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuningcited by 65
  29. 2023Merlin:Empowering Multimodal LLMs with Foresight Mindscited by 46

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 56; 2025: 18; 2026: 10.

84 papers.

Sort:

By area:

  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  2. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  3. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  4. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  5. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 267
  6. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
  7. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  8. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 180
  9. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 174
  10. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 174
  11. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 172
  12. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 169
38 more
  1. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 159
  2. 2025Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distributioncited by 141
  3. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  4. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 114
  5. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  6. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 83
  7. 2024TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergycited by 71
  8. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  9. 2024TextSquare: Scaling up Text-Centric Visual Instruction Tuningcited by 62
  10. 2024Focus Anywhere for Fine-grained Multi-page Document Understandingcited by 62
  11. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  12. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 53
  13. 2024Multimodal Table Understandingcited by 44
  14. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43
  15. 2024UniMERNet: A Universal Network for Real-World Mathematical Expression Recognitioncited by 42
  16. 2024Image Over Text: Transforming Formula Recognition Evaluation with Character Detection Matchingcited by 41
  17. 2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39
  18. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 38
  19. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  20. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  21. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  22. 2024Self-Supervised Visual Preference Alignmentcited by 31
  23. 2024ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Modelscited by 26
  24. 2024Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramidcited by 26
  25. 2024MCTBench: Multimodal Cognition towards Text-Rich Visual Scenes Benchmarkcited by 26
  26. 2024Slow Perception: Let's Perceive Geometric Figures Step-by-stepcited by 24
  27. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  28. 2024Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancementcited by 23
  29. 2025Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAGcited by 23
  30. 2024FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compressioncited by 22
  31. 2024Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plotscited by 21
  32. 2024RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulationcited by 20
  33. 2024Visual Perception by Large Language Model's Weightscited by 19
  34. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  35. 2024Law of Vision Representation in MLLMscited by 18
  36. 2024POINTS: Improving Your Vision-language Model with Affordable Strategiescited by 17
  37. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 17
  38. 2024UNIT: Unifying Image and Text Recognition in One Vision Encodercited by 12

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  2. 2025LaVi: Efficient Large Vision-Language Models via Internal Feature Modulation
  3. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  4. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  5. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model
  6. 2025Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward
  7. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  8. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey