# Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training

arXiv: https://arxiv.org/abs/2509.26625 · HTML: https://mlsanity.com/paper/2509.26625/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 90 of its 106 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Builds on (most cited first, 30 of 90)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)

## Built on by (most cited first, 23 of 23)

- [Large Video Planner Enables Generalizable Robot Control](https://mlsanity.com/paper/2512.15840/) (2025)
- [Beyond Language Modeling: An Exploration of Multimodal Pretraining](https://mlsanity.com/paper/2603.03276/) (2026)
- [VGGT-Omega](https://mlsanity.com/paper/2605.15195/) (2026)
- [MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs](https://mlsanity.com/paper/2511.14159/) (2025)
- [LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs](https://mlsanity.com/paper/2602.00462/) (2026)
- [Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformers](https://mlsanity.com/paper/2511.13945/) (2025)
- [What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models](https://mlsanity.com/paper/2603.00510/) (2026)
- [Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policy](https://mlsanity.com/paper/2601.06801/) (2026)
- [Exploring MLLM-Diffusion Information Transfer with MetaCanvas](https://mlsanity.com/paper/2512.11464/) (2025)
- [RL makes MLLMs see better than SFT](https://mlsanity.com/paper/2510.16333/) (2025)
- [Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes](https://mlsanity.com/paper/2608.05000/) (2026)
- [Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models](https://mlsanity.com/paper/2511.17487/) (2025)
- [From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Images](https://mlsanity.com/paper/2511.22805/) (2025)
- [Linking Perception, Confidence and Accuracy in MLLMs](https://mlsanity.com/paper/2603.12149/) (2026)
- [Beyond Weight Adaptation: Feature-Space Domain Injection for Cross-Modal Ship Re-Identification](https://mlsanity.com/paper/2512.20892/) (2025)
- [Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning](https://mlsanity.com/paper/2602.14518/) (2026)
- [MIRROR: Aligning Semantic Relations from Language to Image via Gromov--Wasserstein](https://mlsanity.com/paper/2606.29462/) (2026)
- [Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation](https://mlsanity.com/paper/2606.29805/) (2026)
- [RADAR: Revealing Asymmetric Development of Abilities in MLLM Pre-training](https://mlsanity.com/paper/2602.12892/) (2026)
- [Learning to Draw ASCII Improves Spatial Reasoning in Language Models](https://mlsanity.com/paper/2604.14641/) (2026)
- [MentalThink: Shaping Thoughts in Mental SVG World](https://mlsanity.com/paper/2607.03530/) (2026)
- [DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes](https://mlsanity.com/paper/2607.24516/) (2026)
- [Procedural Core: A Compact Recurrent Initialization for Vision Transformers](https://mlsanity.com/paper/2609.37631/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs), despite being trained on text alone, surprisingly develop rich visual priors. These priors allow latent visual capabilities to be unlocked for vision tasks with a relatively small amount of multimodal data, and in some cases, to perform visual tasks without ever having seen an image. Through systematic analysis, we reveal that visual priors-the implicit, emergent knowledge about the visual world acquired during language pre-training-are composed of separable perception and reasoning priors with unique scaling trends and origins. We show that an LLM's latent visual reasoning ability is predominantly developed by pre-training on reasoning-centric data (e.g., code, math, academia) and scales progressively. This reasoning prior acquired from language pre-training is transferable and universally applicable to visual reasoning. In contrast, a perception prior emerges more diffusely from broad corpora, and perception ability is more sensitive to the vision encoder and visual instruction tuning data. In parallel, text describing the visual world proves crucial, though its performance impact saturates rapidly. Leveraging these insights, we propose a data-centric recipe for pre-training vision-aware LLMs and verify it in 1T token scale pre-training. Our findings are grounded in over 100 controlled experiments consuming 500,000 GPU-hours, spanning the full MLLM construction pipeline-from LLM pre-training to visual alignment and supervised multimodal fine-tuning-across five model scales, a wide range of data categories and mixtures, and multiple adaptation setups. Along with our main findings, we propose and investigate several hypotheses, and introduce the Multi-Level Existence Bench (MLE-Bench). Together, this work provides a new way of deliberately cultivating visual priors from language pre-training, paving the way for the next generation of multimodal LLMs.