mlsanityData to 7 Oct 2026

LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images

arXiv 2403.11703 ↗First posted 18 Mar 2024cs.CV cs.AIcited by 193

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Visual encoding constitutes the basis of large multimodal models (LMMs) in understanding the visual world. Conventional LMMs process images in fixed sizes and limited resolutions, while recent explorations in this direction are limited in adaptivity, efficiency, and even correctness. In this work, we first take GPT-4V and LLaVA-1.5 as representative examples and expose systematic flaws rooted in their visual encoding strategy. To address the challenges, we present LLaVA-UHD, a large multimodal model that can efficiently perceive images in any aspect ratio and high resolution. LLaVA-UHD includes three key components: (1) An image modularization strategy that divides native-resolution images into smaller variable-sized slices for efficient and extensible encoding, (2) a compression module that further condenses image tokens from visual encoders, and (3) a spatial schema to organize slice tokens for LLMs. Comprehensive experiments show that LLaVA-UHD outperforms established LMMs trained with 2-3 orders of magnitude more data on 9 benchmarks. Notably, our model built on LLaVA-1.5 336x336 supports 6 times larger (i.e., 672x1088) resolution images using only 94% inference computation, and achieves 6.4 accuracy improvement on TextVQA. Moreover, the model can be efficiently trained in academic settings, within 23 hours on 8 A100 GPUs (vs. 26 hours of LLaVA-1.5). We make the data and code publicly available at https://github.com/thunlp/LLaVA-UHD.

Key facts. Cited by 193 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 42 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

36 of 42 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
24 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  7. 2019Towards VQA Models That Can Readcited by 1,696
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  11. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  12. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  13. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  14. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  15. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  16. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  17. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  18. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  19. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  20. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  21. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 112
  22. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  23. 2023OtterHD: A High-Resolution Multi-modality Modelcited by 72
  24. 2024InfiMM-HD: A Leap Forward in High-Resolution Multimodal Understandingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 71; 2025: 78; 2026: 44.

193 papers.

Sort:

By area:

  1. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  3. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  4. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  5. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  6. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  7. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  8. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  9. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  10. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  11. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
  12. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
38 more
  1. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  2. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  3. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  4. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
  5. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  6. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  7. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generationcited by 118
  8. 2025Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Searchcited by 116
  9. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112
  10. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 110
  11. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AIcited by 96
  12. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  13. 2025Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMscited by 94
  14. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 92
  15. 2024Unveiling Encoder-Free Vision-Language Modelscited by 90
  16. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  17. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 82
  18. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 81
  19. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 75
  20. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  21. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perceptioncited by 72
  22. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  23. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  24. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 64
  25. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 59
  26. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  27. 2025XLRS-Bench: Could Your Multimodal LLMs Understand Extremely Large Ultra-High-Resolution Remote Sensing Imagery?cited by 50
  28. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  29. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 45
  30. 2025GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolutioncited by 41
  31. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  32. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Modelscited by 38
  33. 2024SOLO: A Single Transformer for Scalable Vision-Language Modelingcited by 35
  34. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  35. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  36. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  37. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 30
  38. 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understandingcited by 30

50 most cited of 193 All 193 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token
  2. 2025Rethinking Visual Information Processing in Multimodal LLMs
  3. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
  4. 2024AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularity
  5. 2024ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models
  6. 2024LLaVA-OneVision: Easy Visual Task Transfer
  7. 2024iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models
  8. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models