# LOCR: Location-Guided Transformer for Optical Character Recognition

arXiv: https://arxiv.org/abs/2403.02127 · HTML: https://mlsanity.com/paper/2403.02127/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Builds on (most cited first, 22 of 22)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression](https://mlsanity.com/paper/1911.08287/) (2019)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [EAST: An Efficient and Accurate Scene Text Detector](https://mlsanity.com/paper/1704.03155/) (2017)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [Unified Structure Generation for Universal Information Extraction](https://mlsanity.com/paper/2203.12277/) (2022)
- [Scene Text Recognition with Permuted Autoregressive Sequence Models](https://mlsanity.com/paper/2207.06966/) (2022)
- [TableFormer: Robust Transformer Modeling for Table-Text Encoding](https://mlsanity.com/paper/2203.00274/) (2022)
- [Visual In-Context Prompting](https://mlsanity.com/paper/2311.13601/) (2023)
- [Rethinking Text Line Recognition Models](https://mlsanity.com/paper/2104.07787/) (2021)
- [PP-StructureV2: A Stronger Document Analysis System](https://mlsanity.com/paper/2210.05391/) (2022)
- [T-Rex: Counting by Visual Prompting](https://mlsanity.com/paper/2311.13596/) (2023)
- [DocBed: A Multi-Stage OCR Solution for Documents with Complex Layouts](https://mlsanity.com/paper/2202.01414/) (2022)
- [EfficientOCR: An Extensible, Open-Source Package for Efficiently Digitizing World Knowledge](https://mlsanity.com/paper/2310.10050/) (2023)

## Built on by (most cited first, 3 of 3)

- [Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction](https://mlsanity.com/paper/2410.21169/) (2024)
- [μgat: Improving Single-Page Document Parsing by Providing Multi-Page Context](https://mlsanity.com/paper/2408.15646/) (2024)
- [Layout-Aware Text Editing for Efficient Transformation of Academic PDFs to Markdown](https://mlsanity.com/paper/2512.18115/) (2025)

## Abstract (arXiv)

Academic documents are packed with texts, equations, tables, and figures, requiring comprehensive understanding for accurate Optical Character Recognition (OCR). While end-to-end OCR methods offer improved accuracy over layout-based approaches, they often grapple with significant repetition issues, especially with complex layouts in Out-Of-Domain (OOD) documents.To tackle this issue, we propose LOCR, a model that integrates location guiding into the transformer architecture during autoregression. We train the model on a dataset comprising over 77M text-location pairs from 125K academic document pages, including bounding boxes for words, tables and mathematical symbols. LOCR adeptly handles various formatting elements and generates content in Markdown language. It outperforms all existing methods in our test set constructed from arXiv, as measured by edit distance, BLEU, METEOR and F-measure.LOCR also reduces repetition frequency from 4.4% of pages to 0.5% in the arXiv dataset, from 13.2% to 1.3% in OOD quantum physics documents and from 8.1% to 1.8% in OOD marketing documents. Additionally, LOCR features an interactive OCR mode, facilitating the generation of complex documents through a few location prompts from human.