mlsanityData to 14 Oct 2026

OtterHD: A High-Resolution Multi-modality Model

arXiv 2311.04219 ↗First posted 7 Nov 2023cs.CV cs.AIcited by 72

Not on the map. Why?

Cited by 52 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we present OtterHD-8B, an innovative multimodal model evolved from Fuyu-8B, specifically engineered to interpret high-resolution visual inputs with granular precision. Unlike conventional models that are constrained by fixed-size vision encoders, OtterHD-8B boasts the ability to handle flexible input dimensions, ensuring its versatility across various inference requirements. Alongside this model, we introduce MagnifierBench, an evaluation framework designed to scrutinize models' ability to discern minute details and spatial relationships of small objects. Our comparative analysis reveals that while current leading models falter on this benchmark, OtterHD-8B, particularly when directly processing high-resolution inputs, outperforms its counterparts by a substantial margin. The findings illuminate the structural variances in visual information processing among different models and the influence that the vision encoders' pre-training resolution disparities have on model effectiveness within such benchmarks. Our study highlights the critical role of flexibility and high-resolution input capabilities in large multimodal models and also exemplifies the potential inherent in the Fuyu architecture's simplicity for handling complex visual data.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 7 Nov 2023.

Builds on

41 of 55 references.

References · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
29 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  2. 2015VQA: Visual Question Answeringcited by 2,506
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  6. 2019Towards VQA Models That Can Readcited by 1,718
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  11. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  13. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  14. 2016Modeling Context in Referring Expressionscited by 1,063
  15. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  16. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  17. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  18. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  19. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  20. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  21. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  22. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  23. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  24. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  25. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 225
  26. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  27. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 126
  28. 2023VPGTrans: Transfer Visual Prompt Generator across LLMscited by 80
  29. 2022Open-Set Semi-Supervised Object Detectioncited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 47; 2025: 15; 2026: 4.

72 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  3. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  4. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  5. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  6. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  7. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 294
  8. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 220
  9. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
  10. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  11. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 172
  12. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
38 more
  1. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 114
  2. 2024Unveiling Encoder-Free Vision-Language Modelscited by 94
  3. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 92
  4. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 76
  5. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  6. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Modelscited by 75
  7. 2024CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automationcited by 70
  8. 2024Dense Connector for MLLMscited by 49
  9. 2025Similarity-Aware Token Pruning: Your VLM but Fastercited by 35
  10. 2024In-Context Symbolic Regression: Leveraging Large Language Models for Function Discoverycited by 31
  11. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  12. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  13. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  14. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 29
  15. 2024Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referringcited by 29
  16. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  17. 2024ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Modelscited by 26
  18. 2024Quality Assessment in the Era of Large Models: A Surveycited by 26
  19. 2023GlitchBench: Can large multimodal models detect video game glitches?cited by 23
  20. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  21. 2024FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compressioncited by 22
  22. 2023T-Rex: Counting by Visual Promptingcited by 20
  23. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17
  24. 2024HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Modelscited by 16
  25. 2024DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Modelscited by 15
  26. 2024TroL: Traversal of Layers for Large Language and Vision Modelscited by 14
  27. 2024InfiMM-HD: A Leap Forward in High-Resolution Multimodal Understandingcited by 11
  28. 2024ActiView: Evaluating Active Perception Ability for Multimodal Large Language Modelscited by 11
  29. 2024FlexAttention for Efficient High-Resolution Vision-Language Modelscited by 9
  30. 2024AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularitycited by 9
  31. 2024Dragonfly: Multi-Resolution Zoom-In Encoding Enhances Vision-Language Modelscited by 8
  32. 2024Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decodingcited by 8
  33. 2024LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformercited by 8
  34. 20254D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object Understandingcited by 8
  35. 2025LLaVA-SP: Enhancing Visual Representation with Visual Spatial Tokens for MLLMscited by 8
  36. 2024Fact :Teaching MLLMs with Faithful, Concise and Transferable Rationalescited by 7
  37. 2025Enhancing Multimodal In-Context Learning for Image Classification through Coreset Optimizationcited by 7
  38. 2024GeNet: A Multimodal LLM-Based Co-Pilot for Network Topology and Configurationcited by 6

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models
  2. 2025Rethinking Visual Information Processing in Multimodal LLMs
  3. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
  4. 2026BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception
  5. 2024VisualRWKV-HD and UHD: Advancing High-Resolution Processing for Visual Language Models
  6. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Models
  7. 2025OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning
  8. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models