mlsanityData to 7 Oct 2026

FineViT: Progressively Unlocking Fine-Grained Perception with Dense Recaptions

arXiv 2603.17326 ↗First posted 18 Mar 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

While Multimodal Large Language Models (MLLMs) have experienced rapid advancements, their visual encoders frequently remain a performance bottleneck. Conventional CLIP-based encoders struggle with dense spatial tasks due to the loss of visual details caused by low-resolution pretraining and the reliance on noisy, coarse web-crawled image-text pairs. To overcome these limitations, we introduce FineViT, a novel vision encoder specifically designed to unlock fine-grained perception. By replacing coarse web data with dense recaptions, we systematically mitigate information loss through a progressive training paradigm.: first, the encoder is trained from scratch at a high native resolution on billions of global recaptioned image-text pairs, establishing a robust, detail rich semantic foundation. Subsequently, we further enhance its local perception through LLM alignment, utilizing our curated FineCap-450M dataset that comprises over 450 million high quality local captions. Extensive experiments validate the effectiveness of the progressive strategy. FineViT achieves state-of-the-art zero-shot recognition and retrieval performance, especially in long-context retrieval, and consistently outperforms multimodal visual encoders such as SigLIP2 and Qwen-ViT when integrated into MLLMs. We hope FineViT could serve as a powerful new baseline for fine-grained visual perception.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 62 of its 68 references that are in the map. First posted on arXiv 18 Mar 2026.

Builds on

62 of 68 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs23 Oct
  2. 2025FineVision: Open Data Is All You Need20 Oct
  3. 2025Seed1.5-VL Technical Report11 May
  4. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  6. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark14 Feb

Other references · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2025Qwen2.5-VL Technical Reportcited by 5,418
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2025Qwen3-VL Technical Reportcited by 2,534
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
38 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2019Towards VQA Models That Can Readcited by 1,696
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  9. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  11. 2025DINOv3cited by 1,297
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  13. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  14. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  15. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  16. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  17. 2016Modeling Context in Referring Expressionscited by 1,054
  18. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  19. 2025Qwen-Image Technical Reportcited by 973
  20. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  21. 2016A Diagram Is Worth A Dozen Imagescited by 793
  22. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  23. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  24. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  25. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  26. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  27. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  28. 2021InfographicVQAcited by 430
  29. 2020Are we done with ImageNet?cited by 329
  30. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  31. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  32. 2021Learning To Count Everythingcited by 149
  33. 2025PaddleOCR 3.0 Technical Reportcited by 136
  34. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 108
  35. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  36. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  37. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  38. 2022MVP: Multimodality-guided Visual Pre-trainingcited by 92

50 most cited of 56 All 56 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Infusing fine-grained visual knowledge to Vision-Language Models
  2. 2025FG-CLIP: Fine-Grained Visual and Textual Alignment
  3. 2024Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Models
  4. 2026Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models
  5. 2025Perception Encoder: The best visual embeddings are not at the output of the network
  6. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  7. 2026Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis
  8. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception