mlsanityData to 7 Oct 2026

Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models

arXiv 2311.06607 ↗First posted 11 Nov 2023cs.CV cs.AI cs.CLcited by 356

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large Multimodal Models (LMMs) have shown promise in vision-language tasks but struggle with high-resolution input and detailed scene understanding. Addressing these challenges, we introduce Monkey to enhance LMM capabilities. Firstly, Monkey processes input images by dividing them into uniform patches, each matching the size (e.g., 448x448) used in the original training of the well-trained vision encoder. Equipped with individual adapter for each patch, Monkey can handle higher resolutions up to 1344x896 pixels, enabling the detailed capture of complex visual information. Secondly, it employs a multi-level description generation method, enriching the context for scene-object associations. This two-part strategy ensures more effective learning from generated data: the higher resolution allows for a more detailed capture of visuals, which in turn enhances the effectiveness of comprehensive descriptions. Extensive ablative results validate the effectiveness of our designs. Additionally, experiments on 18 datasets further demonstrate that Monkey surpasses existing LMMs in many tasks like Image Captioning and various Visual Question Answering formats. Specially, in qualitative tests focused on dense text question answering, Monkey has exhibited encouraging results compared with GPT4V. Code is available at https://github.com/Yuliang-Liu/Monkey.

Key facts. Cited by 356 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 73 references that are in the map. First posted on arXiv 11 Nov 2023.

Builds on

57 of 73 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  2. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  8. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  10. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Segment Anythingcited by 8,158
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
35 more
  1. 2019Towards VQA Models That Can Readcited by 1,696
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  4. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  11. 2016A Diagram Is Worth A Dozen Imagescited by 793
  12. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  13. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  14. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  15. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  16. 2021InfographicVQAcited by 430
  17. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  18. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  19. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  20. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  21. 2019Scene Text Visual Question Answeringcited by 327
  22. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  23. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  24. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  25. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  26. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  27. 2020PP-OCR: A Practical Ultra Lightweight OCR Systemcited by 147
  28. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  29. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  30. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  31. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  32. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  33. 2021Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layoutscited by 82
  34. 2020On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answeringcited by 78
  35. 2023OtterHD: A High-Resolution Multi-modality Modelcited by 72

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 161; 2025: 131; 2026: 59.

356 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  7. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  8. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  9. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  10. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  11. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  12. 2024What matters when building vision-language models?cited by 321
38 more
  1. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  2. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  3. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  4. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  5. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  6. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  7. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  8. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  9. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  10. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  11. 2024An Introduction to Vision-Language Modelingcited by 175
  12. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  13. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
  14. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  15. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  16. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  17. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  18. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  19. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  20. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  21. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  22. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  23. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  24. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  25. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  26. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  27. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 127
  28. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Modelscited by 125
  29. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  30. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112
  31. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 110
  32. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  33. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  34. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 101
  35. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  36. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  37. 2024Unveiling Encoder-Free Vision-Language Modelscited by 90
  38. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89

50 most cited of 356 All 356 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  3. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images
  4. 2026VisionPangu: A Compact and Fine-Grained Multimodal Assistant with 1.7B Parameters
  5. 2025Visual Representation Alignment for Multimodal Large Language Models
  6. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models
  7. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  8. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies