# Scaling Vision Pre-Training to 4K Resolution

arXiv: https://arxiv.org/abs/2503.19903 · HTML: https://mlsanity.com/paper/2503.19903/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 108 of its 127 references that are in the map.
- First posted on arXiv 25 Mar 2025.

## Builds on (most cited first, 30 of 108)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)

## Built on by (most cited first, 28 of 28)

- [Scaling RL to Long Videos](https://mlsanity.com/paper/2507.07966/) (2025)
- [OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM](https://mlsanity.com/paper/2510.15870/) (2025)
- [High-Resolution Visual Reasoning via Multi-Turn Grounding-Based Reinforcement Learning](https://mlsanity.com/paper/2507.05920/) (2025)
- [A Benchmark for Ultra-High-Resolution Remote Sensing MLLMs](https://mlsanity.com/paper/2512.17319/) (2025)
- [Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perception](https://mlsanity.com/paper/2509.16944/) (2025)
- [Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing](https://mlsanity.com/paper/2603.12254/) (2026)
- [CropVLM: Learning to Zoom for Fine-Grained Vision-Language Perception](https://mlsanity.com/paper/2511.19820/) (2025)
- [HiVLA: A Visual-Grounded-Centric Hierarchical Embodied Manipulation System](https://mlsanity.com/paper/2604.14125/) (2026)
- [Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models](https://mlsanity.com/paper/2604.06912/) (2026)
- [EvoPrune: Early-Stage Visual Token Pruning for Efficient MLLMs](https://mlsanity.com/paper/2603.03681/) (2026)
- [VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes](https://mlsanity.com/paper/2509.25339/) (2025)
- [PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies](https://mlsanity.com/paper/2510.16505/) (2025)
- [FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning](https://mlsanity.com/paper/2510.21311/) (2025)
- [GeoViS: Geospatially Rewarded Visual Search for Remote Sensing Visual Grounding](https://mlsanity.com/paper/2512.02715/) (2025)
- [Fine-grained Token Allocation Via Operation Pruning for Efficient MLLMs](https://mlsanity.com/paper/2507.02909/) (2025)
- [Head-Aware Visual Cropping: Enhancing Fine-Grained VQA with Attention-Guided Subimage](https://mlsanity.com/paper/2601.22483/) (2026)
- [FOVI: A biologically-inspired foveated interface for deep vision models](https://mlsanity.com/paper/2602.03766/) (2026)
- [UHR-BAT: Budget-Aware Token Compression Vision-Language model for Ultra-High-Resolution Remote Sensing](https://mlsanity.com/paper/2604.13565/) (2026)
- [4KLSDB: A Large-Scale Dataset for 4K Image Restoration and Generation](https://mlsanity.com/paper/2605.24762/) (2026)
- [Region-Level Policy Optimization for Fine-grained MLLM Perception](https://mlsanity.com/paper/2609.19745/) (2026)
- [Reg4Pru: Regularisation Through Random Token Routing for Token Pruning](https://mlsanity.com/paper/2602.02163/) (2026)
- [Beyond Pixels: Vector-to-Graph Transformation for Reliable Schematic Auditing](https://mlsanity.com/paper/2602.11678/) (2026)
- [FineViT: Progressively Unlocking Fine-Grained Perception with Dense Recaptions](https://mlsanity.com/paper/2603.17326/) (2026)
- [SigLIP-HD by Fine-to-Coarse Supervision](https://mlsanity.com/paper/2607.09488/) (2026)
- [Dynamic Resolution Routing for Efficient Egocentric Grounding](https://mlsanity.com/paper/2608.01638/) (2026)
- [Can't Find Waldo: Evaluating VLMs' Sensitivity to Image Resolution and Detail Level](https://mlsanity.com/paper/2609.31706/) (2026)
- [FlashGaze: Training-Free Multi-Scale Patch Pruning For Efficient Video Understanding](https://mlsanity.com/paper/2610.04225/) (2026)
- [What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies](https://mlsanity.com/paper/2610.15035/) (2026)

## Abstract (arXiv)

High-resolution perception of visual details is crucial for daily tasks. Current vision pre-training, however, is still limited to low resolutions (e.g., 378 x 378 pixels) due to the quadratic cost of processing larger images. We introduce PS3 that scales CLIP-style vision pre-training to 4K resolution with a near-constant cost. Instead of contrastive learning on global image representation, PS3 is pre-trained by selectively processing local regions and contrasting them with local detailed captions, enabling high-resolution representation learning with greatly reduced computational overhead. The pre-trained PS3 is able to both encode the global image at low resolution and selectively process local high-resolution regions based on their saliency or relevance to a text prompt. When applying PS3 to multi-modal LLM (MLLM), the resulting model, named VILA-HD, significantly improves high-resolution visual perception compared to baselines without high-resolution vision pre-training such as AnyRes and S^2 while using up to 4.3x fewer tokens. PS3 also unlocks appealing scaling properties of VILA-HD, including scaling up resolution for free and scaling up test-time compute for better performance. Compared to state of the arts, PS3 and VILA-HD outperform previous vision encoders (e.g., SigLIP2 and Perception Encoder) and MLLMs (e.g., NVILA and Qwen2.5-VL) respectively across multiple benchmarks and achieve better efficiency than latest token pruning approaches. Finally, we find current benchmarks do not require 4K-resolution perception, which motivates us to propose 4KPro, a new benchmark of image QA at 4K resolution, on which VILA-HD outperforms all previous MLLMs, including a 16.1% improvement over GPT-4o and a 7.5% improvement and 1.67x speedup over Qwen2.5-VL.