mlsanityData to 14 Oct 2026

InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD

arXiv 2404.06512 ↗First posted 9 Apr 2024cs.CV cs.CLcited by 159

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The Large Vision-Language Model (LVLM) field has seen significant advancements, yet its progression has been hindered by challenges in comprehending fine-grained visual content due to limited resolution. Recent efforts have aimed to enhance the high-resolution understanding capabilities of LVLMs, yet they remain capped at approximately 1500 x 1500 pixels and constrained to a relatively narrow resolution range. This paper represents InternLM-XComposer2-4KHD, a groundbreaking exploration into elevating LVLM resolution capabilities up to 4K HD (3840 x 1600) and beyond. Concurrently, considering the ultra-high resolution may not be necessary in all scenarios, it supports a wide range of diverse resolutions from 336 pixels to 4K standard, significantly broadening its scope of applicability. Specifically, this research advances the patch division paradigm by introducing a novel extension: dynamic resolution with automatic patch configuration. It maintains the training image aspect ratios while automatically varying patch counts and configuring layouts based on a pre-trained Vision Transformer (ViT) (336 x 336), leading to dynamic training resolution from 336 pixels to 4K standard. Our research demonstrates that scaling training resolution up to 4K HD leads to consistent performance enhancements without hitting the ceiling of potential improvements. InternLM-XComposer2-4KHD shows superb capability that matches or even surpasses GPT-4V and Gemini Pro in 10 of the 16 benchmarks. The InternLM-XComposer2-4KHD model series with 7B parameters are publicly available at https://github.com/InternLM/InternLM-XComposer.

Key facts. Cited by 159 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 82 of its 109 references that are in the map. First posted on arXiv 9 Apr 2024.

Builds on

82 of 109 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  2. 2024DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Models22 Feb
  3. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model29 Jan
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  5. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM11 Dec
  6. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  7. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  8. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  9. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model8 Oct
  10. 2023WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Models21 Aug
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
4 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  2. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  4. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 66

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2015VQA: Visual Question Answeringcited by 2,506
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
38 more
  1. 2019Towards VQA Models That Can Readcited by 1,718
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  9. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  10. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  11. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  12. 2016A Diagram Is Worth A Dozen Imagescited by 804
  13. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  14. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  15. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  16. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  17. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  18. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  19. 2018nocaps: novel object captioning at scalecited by 444
  20. 2021InfographicVQAcited by 437
  21. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  22. 2024InternLM2 Technical Reportcited by 397
  23. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  24. 2016Visual Dialogcited by 377
  25. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  26. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  27. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  28. 2019Scene Text Visual Question Answeringcited by 331
  29. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  30. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  31. 2022Visual Spatial Reasoningcited by 293
  32. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  33. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  34. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
  35. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  36. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
  37. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 174
  38. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 89; 2025: 53; 2026: 17.

159 papers.

Sort:

By area:

  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  4. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  6. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  7. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  8. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  9. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  10. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  11. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 234
  12. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
38 more
  1. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 183
  2. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  3. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 173
  4. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 172
  5. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 169
  6. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 161
  7. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  8. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 139
  9. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  10. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
  11. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  12. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 114
  13. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 111
  14. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 102
  15. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 97
  16. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  17. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AIcited by 96
  18. 2024Unveiling Encoder-Free Vision-Language Modelscited by 94
  19. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  20. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 90
  21. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  22. 2024DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMscited by 75
  23. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMscited by 74
  24. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
  25. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locatingcited by 72
  26. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 70
  27. 2024I2EBench: A Comprehensive Benchmark for Instruction-based Image Editingcited by 63
  28. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  29. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 59
  30. 2025GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensingcited by 57
  31. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  32. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Modelscited by 54
  33. 2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversioncited by 53
  34. 2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learningcited by 52
  35. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  36. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 45
  37. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  38. 2024MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversitycited by 38

50 most cited of 159 All 159 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Models
  2. 2024FastVLM: Efficient Vision Encoding for Vision Language Models
  3. 2024VisualRWKV-HD and UHD: Advancing High-Resolution Processing for Visual Language Models
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
  5. 2024Xmodel-VLM: A Simple Baseline for Multimodal Vision Language Model
  6. 2024FILA: Fine-Grained Vision Language Models
  7. 2025LLaVA-UHD v3: Progressive Visual Compression for Efficient Native-Resolution Encoding in MLLMs
  8. 2026Language-Assisted Super-Resolution from Real-World Low-Resolution Patches