mlsanityData to 7 Oct 2026

Internalized Reasoning for Long-Context Visual Document Understanding

arXiv 2604.02371 ↗First posted 31 Mar 2026cs.CV cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Visual long-document understanding is critical for enterprise, legal, and scientific applications, yet the best performing open recipes have not explored reasoning, a capability which has driven leaps in math and code performance. We introduce a synthetic data pipeline for reasoning in long-document understanding that generates thinking traces by scoring each page for question relevance, extracting textual evidence and ordering it from most to least relevant. We apply SFT to the resulting traces within texttt<think> tags, gated by a texttt<cot> control token, and the resulting reasoning capability is internalized via low-strength model merging. We study Qwen3 VL 32B and Mistral Small 3.1 24B. With Qwen3 VL, we achieve 58.3 on MMLongBenchDoc, surpassing the 7× larger Qwen3 VL 235B A22B (57.0). With Mistral, we show that synthetic reasoning outperforms distillation from the Thinking version's traces by 3.8 points on MMLBD-C, and internalized reasoning exhibits 12.4× fewer mean output tokens compared to explicit reasoning. We release our pipeline for reproducibility and further exploration.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 49 references that are in the map. First posted on arXiv 31 Mar 2026.

Builds on

39 of 49 references.

References · 39

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  4. 2022Editing Models with Task Arithmeticcited by 1,076
  5. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  6. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  7. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  8. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  9. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  10. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  11. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  12. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 256
27 more
  1. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  2. 2021Thinking Like Transformerscited by 182
  3. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  4. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 165
  5. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  6. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 135
  7. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  8. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 72
  9. 2024TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domainscited by 68
  10. 2025QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learningcited by 46
  11. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  12. 2026Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thoughtcited by 35
  13. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughlycited by 29
  14. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  15. 2025Temporal Chain of Thought: Long-Video Understanding by Thinking in Framescited by 27
  16. 2025LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimizationcited by 24
  17. 2025NExtLong: Toward Effective Long-Context Training without Long Documentscited by 19
  18. 2025LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Datacited by 16
  19. 2024Investigating Mysteries of CoT-Augmented Distillationcited by 15
  20. 2024Distilling Reasoning Ability from Large Language Models with Adaptive Thinkingcited by 13
  21. 2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimizationcited by 10
  22. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  23. 2025Deep Thinking by Markov Chain of Continuous Thoughtscited by 8
  24. 2025Chain-of-Thought Tokens are Computer Program Variablescited by 5
  25. 2025Dual-Head Reasoning Distillation: Improving Classifier Accuracy with Train-Time-Only Reasoningcited by 3
  26. 2025Luth: Efficient French Specialization for Small Language Models and Cross-Lingual Transfercited by 3
  27. 2026How to Train Your Long-Context Visual Document Modelcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026DocTrace: Towards Traceable Long Document VQA via Hierarchical Evidence Graph Reasoning
  2. 2026Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA
  3. 2026SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding
  4. 2026DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents
  5. 2026π2: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models
  6. 2026DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding
  7. 2026SEER: Long-Context Reasoning via Selective Visual-Text Compression
  8. 2026TestHallVQA: Exploring LVLMs' Document-Level Reasoning under Redundant Contexts from Scientific Exams