One Patch Is Enough: Reinforcement-Optimized Visual Token Grounding for MLLM-Based Scene Text Spotting
Not on the map. Why?
Abstract (arXiv)
Scene text spotting requires high-precision alignment between textual recognition and spatial localization. While visual-token grounding has emerged as a promising formulation for Multimodal Large Language Models (MLLMs), the previous multi-patch paradigm often introduces redundant noise and localization ambiguity, particularly for dense or small text instances. To address this, we propose Single-Patch Text Spotting (SPaTS), a vision-centric framework that routes each text instance through a single anchor visual token and then recovers geometry via full-image refinement. To accurately identify this anchor without oracle labels, we introduce Single-Patch Selective Optimization (SPaSO), a reinforcement learning framework that optimizes discrete visual-token selection using patch-level rewards. To further improve representation robustness and localization precision, we introduce Directional Embedding Alignment (DEA) to suppress unstable norm bias by decoupling feature magnitude and direction, and Patch-Enhanced Decoding (PED) to fuse the routed anchor with language semantics and cross-attend over the full-image feature map for geometry-aware boundary regression beyond coordinate-space surrogates. Extensive experiments demonstrate that SPaTS consistently and significantly outperforms both frontier closed-source MLLMs and OCR MLLMs. Code is available at https://github.com/eeNickTang/SPaTS.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 54 references that are in the map. First posted on arXiv 30 Jul 2026.
Builds on
23 of 54 references.
References · 23
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
- 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
- 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
- 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
- 2016Rationalizing Neural Predictionscited by 281
- 2019Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapescited by 171
- 2020ABCNet: Real-time Scene Text Spotting with Adaptive Bezier-Curve Networkcited by 170
- 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
- 2017Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognitioncited by 161
- 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
- 2018Scene Text Detection and Recognition: The Deep Learning Eracited by 114
11 more
- 2018FOTS: Fast Oriented Text Spotting with a Unified Networkcited by 109
- 2022Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusioncited by 94
- 2021ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spottingcited by 84
- 2022DeepSolo: Let Transformer Decoder with Explicit Points Solo for Text Spottingcited by 74
- 2023SPTS v2: Single-Point Scene Text Spottingcited by 66
- 2024VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Drivingcited by 49
- 2025Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMscited by 20
- 2026Multimodal OCR: Parse Anything from Documentscited by 12
- 2025OCRGenBench: A Comprehensive Benchmark for Evaluating OCR Generative Capabilitiescited by 8
- 2024ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehensioncited by 6
- 2025Smaller But Better: Unifying Layout Generation with Smaller Large Language Modelscited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026SupGRPO: Enhancing GRPO with Matching-based Online SFT for Text Spotting
- 2026Layer-Aware Position Embeddings for Visual Token Pruning in Multimodal Large Language Models
- 2025FlashVLM: Text-Guided Visual Token Selection for Large Multimodal Models
- 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
- 2025REVEALER: Reinforcement-Guided Visual Reasoning for Element-Level Text-Image Alignment Evaluation
- 2026MEVL-STP: Multi-Encoder and Vision Language Model for Arbitrarily Shaped Scene Text Spotting
- 2026One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs
- 2026TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment