# Think with Structured Grounding: Perceptual Reinforcement Learning for Chart and Visual-Tabular Understanding

arXiv: https://arxiv.org/abs/2608.22429 · HTML: https://mlsanity.com/paper/2608.22429/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 30 of its 65 references that are in the map.
- First posted on arXiv 23 Aug 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 30)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning](https://mlsanity.com/paper/2504.08837/) (2025)
- [MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning](https://mlsanity.com/paper/2503.07365/) (2025)
- [LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training](https://mlsanity.com/paper/2509.23661/) (2025)
- [MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe](https://mlsanity.com/paper/2509.18154/) (2025)
- [Thyme: Think Beyond Images](https://mlsanity.com/paper/2508.11630/) (2025)
- [ChartLlama: A Multimodal LLM for Chart Understanding and Generation](https://mlsanity.com/paper/2311.16483/) (2023)
- [Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learning](https://mlsanity.com/paper/2505.14677/) (2025)
- [TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domains](https://mlsanity.com/paper/2404.19205/) (2024)
- [Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality](https://mlsanity.com/paper/2505.18227/) (2025)
- [Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception](https://mlsanity.com/paper/2602.11858/) (2026)
- [Chart-R1: Chain-of-Thought Supervision and Reinforcement for Advanced Chart Reasoner](https://mlsanity.com/paper/2507.15509/) (2025)
- [Interleaved Reasoning for Large Language Models via Reinforcement Learning](https://mlsanity.com/paper/2505.19640/) (2025)
- [M2-Reasoning: Empowering MLLMs with Unified General and Spatial Reasoning](https://mlsanity.com/paper/2507.08306/) (2025)
- [ChartQA-X: Generating Explanations for Visual Chart Reasoning](https://mlsanity.com/paper/2504.13275/) (2025)
- [VACoT: Rethinking Visual Data Augmentation with VLMs](https://mlsanity.com/paper/2512.02361/) (2025)
- [PatchCue: Enhancing Vision-Language Model Reasoning with Patch-Based Visual Cues](https://mlsanity.com/paper/2603.05869/) (2026)
- [Venus-DeFakerOne: Unified Fake Image Detection & Localization](https://mlsanity.com/paper/2605.14091/) (2026)
- [Chart-RVR: Reinforcement Learning with Verifiable Rewards for Explainable Chart Reasoning](https://mlsanity.com/paper/2510.10973/) (2025)
- [Beyond Human Annotation: Recent Advances in Data Generation Methods for Document Intelligence](https://mlsanity.com/paper/2601.12318/) (2026)
- [HyTRec: A Hybrid Temporal-Aware Attention Architecture for Long Behavior Sequential Recommendation](https://mlsanity.com/paper/2602.18283/) (2026)
- [FakeVLM-R1: Internalizing Physical Laws via CoT for Synthetic Image Detection](https://mlsanity.com/paper/2605.30062/) (2026)
- [Know What You Know: Metacognitive Entropy Calibration for Verifiable RL Reasoning](https://mlsanity.com/paper/2602.22751/) (2026)
- [Veritas++: Value-aware On-Policy Distillation for Perception-Enhanced AIGI Detection](https://mlsanity.com/paper/2607.27113/) (2026)
- [UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models](https://mlsanity.com/paper/2608.08627/) (2026)

## Built on by (most cited first, 2 of 2)

- [UniAR: A Unified Framework for Autism Recognition Enhanced by Multi-View Prompt Learning](https://mlsanity.com/paper/2609.31298/) (2026)
- [PolyOCR-Venus: Unified OCR Foundation Models for Text-Centric Visual Intelligence](https://mlsanity.com/paper/2609.37712/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) capable of thinking with images often rely on external tools for fine-grained perception. However, this reliance introduces significant inference latency and fails to effectively resolve the spatial-structural gap-a fundamental challenge in text-dense and structurally relational visuals (e.g., charts and visual tables) where strict relative spatial arrangements bind textual elements. Without external tools, standard MLLMs struggle with such fine-grained visual reasoning tasks. To address these issues, we propose Think with Structured Grounding (TwSG), a novel fine-grained image perception framework designed to internalize complex images's tool-use capabilities within the model. TwSG distills the benefits of multi-step reasoning and micro-cropping into a single efficient forward pass during inference. Specifically, we use an MLLM to identify key regions guided by ground-truth answers, and then prompt a teacher model to generate high-quality visual question-answering (VQA) data. These fine-grained, region-based supervisory signals are subsequently distilled back into the full-image representation. Our training pipeline consists of two stages: (1) a cold-start supervised fine-tuning (SFT) phase using multi-turn data with focused area descriptions to foster complex reasoning and error recovery; and (2) a reinforcement fine-tuning (RFT) phase driven by a novel process reward mechanism, TL-GRPO, which encourages strategic reasoning. Extensive experiments across various MLLM architectures demonstrate that TwSG reduces inference latency while substantially improving accuracy and robustness, endowing models with native fine-grained region description and flexible reasoning capabilities.