mlsanityData to 7 Oct 2026

Small Language Model Meets with Reinforced Vision Vocabulary

arXiv 2401.12503 ↗First posted 23 Jan 2024cs.CVcited by 43

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Playing Large Vision Language Models (LVLMs) in 2023 is trendy among the AI community. However, the relatively large number of parameters (more than 7B) of popular LVLMs makes it difficult to train and deploy on consumer GPUs, discouraging many researchers with limited resources. Imagine how cool it would be to experience all the features of current LVLMs on an old GTX1080ti (our only game card). Accordingly, we present Vary-toy in this report, a small-size Vary along with Qwen-1.8B as the base ``large'' language model. In Vary-toy, we introduce an improved vision vocabulary, allowing the model to not only possess all features of Vary but also gather more generality. Specifically, we replace negative samples of natural images with positive sample data driven by object detection in the procedure of generating vision vocabulary, more sufficiently utilizing the capacity of the vocabulary network and enabling it to efficiently encode visual information corresponding to natural objects. For experiments, Vary-toy can achieve 65.6% ANLS on DocVQA, 59.1% accuracy on ChartQA, 88.1% accuracy on RefCOCO, and 29% on MMVet. The code will be publicly available on the homepage.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 58 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

45 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models11 Dec
  2. 2023NExT-Chat: An LMM for Chat, Detection and Segmentation8 Nov
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  5. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov

Other references · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2023Segment Anythingcited by 8,158
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
28 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2019Towards VQA Models That Can Readcited by 1,696
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  10. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  11. 2018CornerNet: Detecting Objects as Paired Keypointscited by 763
  12. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  13. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  14. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  15. 2021OCR-free Document Understanding Transformercited by 400
  16. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  17. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  18. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  19. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  20. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  21. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  22. 2019Bottom-up Object Detection by Grouping Extreme and Center Pointscited by 156
  23. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  24. 2022Language Models are General-Purpose Interfacescited by 96
  25. 2022End-to-end Document Recognition and Understanding with Dessurtcited by 73
  26. 2021LaTr: Layout-Aware Transformer for Scene-Text VQAcited by 72
  27. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuningcited by 65
  28. 2023Merlin:Empowering Multimodal LLMs with Foresight Mindscited by 46

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 12; 2026: 3.

43 papers.

Sort:

By area:

  1. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  2. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  3. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  4. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 255
  5. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 174
  6. 2025DeepSeek-OCR: Contexts Optical Compressioncited by 174
  7. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  8. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  9. 2024Efficient Multimodal Learning from Data-centric Perspectivecited by 120
  10. 2025Perception-R1: Pioneering Perception Policy with Reinforcement Learningcited by 116
  11. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  12. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
31 more
  1. 2026DeepSeek-OCR 2: Visual Causal Flowcited by 68
  2. 2024Focus Anywhere for Fine-grained Multi-page Document Understandingcited by 62
  3. 2024I2EBench: A Comprehensive Benchmark for Instruction-based Image Editingcited by 61
  4. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  5. 2024Multimodal Table Understandingcited by 43
  6. 2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39
  7. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 37
  8. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  9. 2025Unhackable Temporal Rewarding for Scalable Video MLLMscited by 33
  10. 2024Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramidcited by 26
  11. 2024Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Modelscited by 24
  12. 2024Slow Perception: Let's Perceive Geometric Figures Step-by-stepcited by 24
  13. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 22
  14. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to Seecited by 21
  15. 2024Imp: Highly Capable Large Multimodal Models for Mobile Devicescited by 19
  16. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 17
  17. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  18. 2024READoc: A Unified Benchmark for Realistic Document Structured Extractioncited by 15
  19. 2024MMRo: Are Multimodal LLMs Eligible as the Brain for In-Home Robotics?cited by 14
  20. 2024UNIT: Unifying Image and Text Recognition in One Vision Encodercited by 12
  21. 2025Learning to Inference Adaptively for Multimodal Large Language Modelscited by 12
  22. 2024StrucTexTv3: An Efficient Vision-Language Model for Text-rich Image Perception, Comprehension, and Beyondcited by 11
  23. 2025'Eclair -- Extracting Content and Layout with Integrated Reading Order for Documentscited by 9
  24. 2025HIPPO: Enhancing the Table Understanding Capability of LLMs through Hybrid-Modal Preference Optimizationcited by 5
  25. 2025Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiencycited by 5
  26. 2025Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translationcited by 4
  27. 2024Transformers Utilization in Chart Understanding: A Review of Recent Advances & Future Trendscited by 3
  28. 2024AdaptVision: Dynamic Input Scaling in MLLMs for Versatile Scene Understandingcited by 1
  29. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 1
  30. 2026CoReTab: Improving Multimodal Table Understanding with Code-driven Reasoningcited by 1
  31. 2026Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancementcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025NanoVLMs: How small can we go and still make coherent Vision Language Models?
  2. 2025Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward
  3. 2024LM4LV: A Frozen Large Language Model for Low-level Vision Tasks
  4. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  5. 2025Vision Language Models: A Survey of 26K Papers
  6. 2024What matters when building vision-language models?
  7. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  8. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks