# Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters

arXiv: https://arxiv.org/abs/2411.03312 · HTML: https://mlsanity.com/paper/2411.03312/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 38 of its 52 references that are in the map.
- First posted on arXiv 5 Nov 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 30 of 38)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models](https://mlsanity.com/paper/2403.15388/) (2024)
- [Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models](https://mlsanity.com/paper/2403.18814/) (2024)
- [Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking](https://mlsanity.com/paper/2403.09629/) (2024)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [TokenPacker: Efficient Visual Projector for Multimodal LLM](https://mlsanity.com/paper/2407.02392/) (2024)

## Built on by (most cited first, 25 of 25)

- [Think Twice, Act Once: Token-Aware Compression and Action Reuse for Efficient Inference in Vision-Language-Action Models](https://mlsanity.com/paper/2505.21200/) (2025)
- [TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Models](https://mlsanity.com/paper/2503.10501/) (2025)
- [Toward Cognitive Supersensing in Multimodal Large Language Model](https://mlsanity.com/paper/2602.01541/) (2026)
- [VAGUE: Visual Contexts Clarify Ambiguous Expressions](https://mlsanity.com/paper/2411.14137/) (2024)
- [Limits and Gains of Test-Time Scaling in Vision-Language Reasoning](https://mlsanity.com/paper/2512.11109/) (2025)
- [SpecVLM: Fast Speculative Decoding in Vision-Language Models](https://mlsanity.com/paper/2509.11815/) (2025)
- [Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Models](https://mlsanity.com/paper/2603.05950/) (2026)
- [EarlyTom: Early Token Compression Completes Fast Video Understanding](https://mlsanity.com/paper/2605.30010/) (2026)
- [TAIJI: Textual Anchoring for Immunizing Jailbreak Images in Vision Language Models](https://mlsanity.com/paper/2503.10872/) (2025)
- [Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning](https://mlsanity.com/paper/2505.11945/) (2025)
- [Inference Compute-Optimal Video Vision Language Models](https://mlsanity.com/paper/2505.18855/) (2025)
- [Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models](https://mlsanity.com/paper/2508.06038/) (2025)
- [HIVTP: A Training-Free Method to Improve VLMs Efficiency via Hierarchical Visual Token Pruning Using Middle-Layer-Based Importance Score](https://mlsanity.com/paper/2509.23663/) (2025)
- [Fwd2Bot: LVLM Visual Token Compression with Double Forward Bottleneck](https://mlsanity.com/paper/2503.21757/) (2025)
- [How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A](https://mlsanity.com/paper/2605.16359/) (2026)
- [Visual Token Compression Enhances Robustness of MLLMs](https://mlsanity.com/paper/2607.22716/) (2026)
- [UniCompress: Token Compression for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2603.11320/) (2026)
- [PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understanding](https://mlsanity.com/paper/2605.30126/) (2026)
- [Baseline-Relative Counterfactual Refinement for Bit-Aware Visual Token Communication](https://mlsanity.com/paper/2608.16192/) (2026)
- [Vision Token Masking Alone Cannot Prevent PHI Leakage in Medical Document OCR: A Systematic Evaluation](https://mlsanity.com/paper/2511.18272/) (2025)
- [Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders](https://mlsanity.com/paper/2603.19209/) (2026)
- [ETC: Extreme Token Compression via Task-aware Visual Information Distillation in VLMs](https://mlsanity.com/paper/2606.00543/) (2026)
- [Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligence](https://mlsanity.com/paper/2607.20981/) (2026)
- [From Saliency to Discriminability: Rank-Preserving Visual Token Pruning for VLM Rerankers](https://mlsanity.com/paper/2609.00667/) (2026)
- [Beyond Selection: Token Parameterization for Extreme Visual Token Compression](https://mlsanity.com/paper/2609.35232/) (2026)

## Abstract (arXiv)

Vision Language Models (VLMs) have demonstrated strong capabilities across various visual understanding and reasoning tasks, driven by incorporating image representations into the token inputs of Large Language Models (LLMs). However, their real-world deployment is often constrained by high latency during inference due to the substantial compute required by the LLM to process the large number of input tokens, predominantly arising from the image. To reduce inference costs, one can either downsize the LLM or reduce the number of input tokens needed to represent the image, the latter of which has been the focus of many recent efforts around token compression. However, it is unclear what the optimal trade-off is given a fixed inference budget. We first characterize this optimal trade-off between the number of visual tokens and LLM parameters by establishing scaling laws that capture variations in performance with these two factors. Our results reveal a surprising trend: for visual reasoning tasks, the inference-optimal behavior in VLMs is achieved by using the largest LLM that fits within the inference budget while minimizing visual token count - often to a single token. While the token reduction literature has mainly focused on maintaining base model performance by modestly reducing the token count (e.g., 5-10×), our results indicate that the compute-optimal inference regime requires operating under even higher token compression ratios. Based on these insights, we take the first steps toward designing token compression algorithms tailored for high-compression settings, utilizing prompt-based compression of tokens. Our work underscores the performance and efficiency benefits of operating in low visual token regimes and the importance of developing tailored token reduction algorithms for such conditions. Code is available at https://github.com/locuslab/llava-token-compression.