# GRAM: Global Reasoning for Multi-Page VQA

arXiv: https://arxiv.org/abs/2401.03411 · HTML: https://mlsanity.com/paper/2401.03411/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 33 of its 38 references that are in the map.
- First posted on arXiv 7 Jan 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [LiT: Zero-Shot Transfer with Locked-image text Tuning](https://mlsanity.com/paper/2111.07991/) (2021)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [Scene Text Visual Question Answering](https://mlsanity.com/paper/1905.13648/) (2019)
- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [Unifying Vision, Text, and Layout for Universal Document Processing](https://mlsanity.com/paper/2212.02623/) (2022)
- [Document Understanding Dataset and Evaluation (DUDE)](https://mlsanity.com/paper/2305.08455/) (2023)
- [Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer](https://mlsanity.com/paper/2102.09550/) (2021)
- [Hierarchical multimodal transformers for Multi-Page DocVQA](https://mlsanity.com/paper/2212.05935/) (2022)
- [Scaling Transformer to 1M tokens and beyond with RMT](https://mlsanity.com/paper/2304.11062/) (2023)
- [CoLT5: Faster Long-Range Transformers with Conditional Computation](https://mlsanity.com/paper/2303.09752/) (2023)
- [Sequence-to-Sequence Contrastive Learning for Text Recognition](https://mlsanity.com/paper/2012.10873/) (2020)
- [SCATTER: Selective Context Attentional Scene Text Recognizer](https://mlsanity.com/paper/2003.11288/) (2020)
- [ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding](https://mlsanity.com/paper/2210.06155/) (2022)
- [DocFormerv2: Local Features for Document Understanding](https://mlsanity.com/paper/2306.01733/) (2023)
- [Document Collection Visual Question Answering](https://mlsanity.com/paper/2104.14336/) (2021)
- [Multimodal Semi-Supervised Learning for Text Recognition](https://mlsanity.com/paper/2205.03873/) (2022)

## Built on by (most cited first, 23 of 23)

- [M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding](https://mlsanity.com/paper/2411.04952/) (2024)
- [LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating](https://mlsanity.com/paper/2412.18424/) (2024)
- [DocVLM: Make Your VLM an Efficient Reader](https://mlsanity.com/paper/2412.08746/) (2024)
- [FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understanding](https://mlsanity.com/paper/2504.17447/) (2025)
- [Deep Learning based Visually Rich Document Content Understanding: A Survey](https://mlsanity.com/paper/2408.01287/) (2024)
- [PDF-WuKong: A Large Multimodal Model for Efficient Long PDF Reading with End-to-End Sparse Sampling](https://mlsanity.com/paper/2410.05970/) (2024)
- [Notes on Applicability of GPT-4 to Document Understanding](https://mlsanity.com/paper/2405.18433/) (2024)
- [μgat: Improving Single-Page Document Parsing by Providing Multi-Page Context](https://mlsanity.com/paper/2408.15646/) (2024)
- [DocVXQA: Context-Aware Visual Explanations for Document Question Answering](https://mlsanity.com/paper/2505.07496/) (2025)
- [Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trends](https://mlsanity.com/paper/2501.02235/) (2025)
- [AVIR: Adaptive Visual In-Document Retrieval for Efficient Multi-Page Document Question Answering](https://mlsanity.com/paper/2601.11976/) (2026)
- [Arctic-TILT. Business Document Understanding at Sub-Billion Scale](https://mlsanity.com/paper/2408.04632/) (2024)
- [M-DocSum: Do LVLMs Genuinely Comprehend Interleaved Image-Text in Document Summarization?](https://mlsanity.com/paper/2503.21839/) (2025)
- [Multimodal Adaptive Inference for Document Image Classification with Anytime Early Exiting](https://mlsanity.com/paper/2405.12705/) (2024)
- [Enhancing Document VQA Models via Retrieval-Augmented Generation](https://mlsanity.com/paper/2508.18984/) (2025)
- [ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documents](https://mlsanity.com/paper/2510.25668/) (2025)
- [DocSLM: A Small Vision-Language Model for Long Multimodal Document Understanding](https://mlsanity.com/paper/2511.11313/) (2025)
- [TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Models](https://mlsanity.com/paper/2411.04642/) (2024)
- [MGA-VQA: Secure and Interpretable Graph-Augmented Visual Question Answering with Memory-Guided Protection Against Unauthorized Knowledge Use](https://mlsanity.com/paper/2511.17881/) (2025)
- [Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligence](https://mlsanity.com/paper/2609.32041/) (2026)
- [LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrieval](https://mlsanity.com/paper/2603.26683/) (2026)
- [ColGraphRAG: Late-Interaction Evidence Retrieval for Multimodal GraphRAG](https://mlsanity.com/paper/2607.16208/) (2026)
- [VLD-RAG: Agentic Vision-Language Retrieval-Augmented Generation for Long, Visually-Rich Multi-Page Documents](https://mlsanity.com/paper/2607.24748/) (2026)

## Abstract (arXiv)

The increasing use of transformer-based large language models brings forward the challenge of processing long sequences. In document visual question answering (DocVQA), leading methods focus on the single-page setting, while documents can span hundreds of pages. We present GRAM, a method that seamlessly extends pre-trained single-page models to the multi-page setting, without requiring computationally-heavy pretraining. To do so, we leverage a single-page encoder for local page-level understanding, and enhance it with document-level designated layers and learnable tokens, facilitating the flow of information across pages for global reasoning. To enforce our model to utilize the newly introduced document tokens, we propose a tailored bias adaptation method. For additional computational savings during decoding, we introduce an optional compression stage using our compression-transformer (C-Former),reducing the encoded sequence length, thereby allowing a tradeoff between quality and latency. Extensive experiments showcase GRAM's state-of-the-art performance on the benchmarks for multi-page DocVQA, demonstrating the effectiveness of our approach.