# Introducing Visual Perception Token into Multimodal Large Language Model

arXiv: https://arxiv.org/abs/2502.17425 · HTML: https://mlsanity.com/paper/2502.17425/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 40 of its 49 references that are in the map.
- First posted on arXiv 24 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 30 of 40)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [TextCaps: a Dataset for Image Captioning with Reading Comprehension](https://mlsanity.com/paper/2003.12462/) (2020)
- [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023)
- [SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models](https://mlsanity.com/paper/2406.01584/) (2024)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [Visual Spatial Reasoning](https://mlsanity.com/paper/2205.00363/) (2022)
- [What does CLIP know about a red circle? Visual prompt engineering for VLMs](https://mlsanity.com/paper/2304.06712/) (2023)
- [CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2109.11797/) (2021)
- [LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness](https://mlsanity.com/paper/2409.18125/) (2024)

## Built on by (most cited first, 26 of 26)

- [VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning](https://mlsanity.com/paper/2505.22019/) (2025)
- [ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning](https://mlsanity.com/paper/2510.27492/) (2025)
- [MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning](https://mlsanity.com/paper/2506.05331/) (2025)
- [VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models](https://mlsanity.com/paper/2511.11007/) (2025)
- [Qwen Look Again: Guiding Vision-Language Reasoning Models to Re-attention Visual Information](https://mlsanity.com/paper/2505.23558/) (2025)
- [VeriThinker: Learning to Verify Makes Reasoning Model Efficient](https://mlsanity.com/paper/2505.17941/) (2025)
- [Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perception](https://mlsanity.com/paper/2509.16944/) (2025)
- [CoFFT: Chain of Foresight-Focus Thought for Visual Language Models](https://mlsanity.com/paper/2509.22010/) (2025)
- [VimRAG: Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graph](https://mlsanity.com/paper/2602.12735/) (2026)
- [Improving Visual Reasoning with Iterative Evidence Refinement](https://mlsanity.com/paper/2603.14117/) (2026)
- [Enhancing Spatial Reasoning through Visual and Textual Thinking](https://mlsanity.com/paper/2507.20529/) (2025)
- [Grounding the Score: Explicit Visual Premise Verification for Reliable Vision-Language Process Reward Models](https://mlsanity.com/paper/2603.16253/) (2026)
- [Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs](https://mlsanity.com/paper/2604.12896/) (2026)
- [Visual Enhanced Depth Scaling for Multimodal Latent Reasoning](https://mlsanity.com/paper/2604.10500/) (2026)
- [Blink: Dynamic Visual Token Resolution for Enhanced Multimodal Understanding](https://mlsanity.com/paper/2512.10548/) (2025)
- [iSHIFT: Lightweight Slow-Fast GUI Agent with Adaptive Perception](https://mlsanity.com/paper/2512.22009/) (2025)
- [From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Grounding](https://mlsanity.com/paper/2605.15951/) (2026)
- [ClueAegis: Heuristic-to-Reasoning Cognitive-skill Learning for Unified Evidence-based Synthetic Image Detection](https://mlsanity.com/paper/2605.25009/) (2026)
- [DeltaV: Thinking with Visual State Updates in Unified Large Multimodal Models](https://mlsanity.com/paper/2607.08434/) (2026)
- [Token Warping Helps MLLMs Look from Nearby Viewpoints](https://mlsanity.com/paper/2604.02870/) (2026)
- [GazeVLM: Active Vision via Internal Attention Control for Multimodal Reasoning](https://mlsanity.com/paper/2605.07817/) (2026)
- [ROVER: Routing Object-Centric Visual Evidence for Grounded Multi-Image Reasoning](https://mlsanity.com/paper/2605.27959/) (2026)
- [From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models](https://mlsanity.com/paper/2606.26196/) (2026)
- [Latent Noise Mask for Reducing Visual Redundancy in Multimodal Large Language Models](https://mlsanity.com/paper/2606.30168/) (2026)
- [Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions](https://mlsanity.com/paper/2607.14499/) (2026)
- [RegRet: Enhancing Region-Level Retrieval in Large Multimodal Models](https://mlsanity.com/paper/2609.16847/) (2026)

## Abstract (arXiv)

To utilize visual information, Multimodal Large Language Model (MLLM) relies on the perception process of its vision encoder. The completeness and accuracy of visual perception significantly influence the precision of spatial reasoning, fine-grained understanding, and other tasks. However, MLLM still lacks the autonomous capability to control its own visual perception processes, for example, selectively reviewing specific regions of an image or focusing on information related to specific object categories. In this work, we propose the concept of Visual Perception Token, aiming to empower MLLM with a mechanism to control its visual perception processes. We design two types of Visual Perception Tokens, termed the Region Selection Token and the Vision Re-Encoding Token. MLLMs autonomously generate these tokens, just as they generate text, and use them to trigger additional visual perception actions. The Region Selection Token explicitly identifies specific regions in an image that require further perception, while the Vision Re-Encoding Token uses its hidden states as control signals to guide additional visual perception processes. Extensive experiments demonstrate the advantages of these tokens in handling spatial reasoning, improving fine-grained understanding, and other tasks. On average, the introduction of Visual Perception Tokens improves the performance of a 2B model by 23.6%, increasing its score from 0.572 to 0.708, and even outperforms a 7B parameter model by 13.4% (from 0.624). Please check out our repo https://github.com/yu-rp/VisualPerceptionToken