# Block-level Text Spotting with LLMs

arXiv: https://arxiv.org/abs/2406.13208 · HTML: https://mlsanity.com/paper/2406.13208/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 19 Jun 2024.

## Builds on (most cited first, 18 of 18)

- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Scene Text Recognition with Permuted Autoregressive Sequence Models](https://mlsanity.com/paper/2207.06966/) (2022)
- [Chain of Code: Reasoning with a Language Model-Augmented Code Emulator](https://mlsanity.com/paper/2312.04474/) (2023)
- [Towards End-to-End Unified Scene Text Detection and Layout Analysis](https://mlsanity.com/paper/2203.15143/) (2022)
- [LayoutReader: Pre-training of Text and Layout for Reading Order Detection](https://mlsanity.com/paper/2108.11591/) (2021)
- [OrigamiNet: Weakly-Supervised, Segmentation-Free, One-Step, Full Page Text Recognition by learning to unfold](https://mlsanity.com/paper/2006.07491/) (2020)
- [Joint Line Segmentation and Transcription for End-to-End Handwritten Paragraph Recognition](https://mlsanity.com/paper/1604.08352/) (2016)
- [Scan, Attend and Read: End-to-End Handwritten Paragraph Recognition with MDLSTM Attention](https://mlsanity.com/paper/1604.03286/) (2016)
- [ICDAR 2023 Competition on Hierarchical Text Detection and Recognition](https://mlsanity.com/paper/2305.09750/) (2023)
- [SPAN: a Simple Predict & Align Network for Handwritten Paragraph Recognition](https://mlsanity.com/paper/2102.08742/) (2021)
- [Post-OCR Paragraph Recognition by Graph Convolutional Networks](https://mlsanity.com/paper/2101.12741/) (2021)
- [Improving Length-Generalization in Transformers via Task Hinting](https://mlsanity.com/paper/2310.00726/) (2023)
- [A Computationally Efficient Pipeline Approach to Full Page Offline Handwritten Text Recognition](https://mlsanity.com/paper/1910.00663/) (2019)
- [Exploring and Improving the Spatial Reasoning Abilities of Large Language Models](https://mlsanity.com/paper/2312.01054/) (2023)
- [Hierarchical Text Spotter for Joint Text Spotting and Layout Analysis](https://mlsanity.com/paper/2310.17674/) (2023)
- [Unified Line and Paragraph Detection by Graph Convolutional Networks](https://mlsanity.com/paper/2203.09638/) (2022)
- [Contextual Text Block Detection towards Scene Text Understanding](https://mlsanity.com/paper/2207.12955/) (2022)

## Abstract (arXiv)

Text spotting has seen tremendous progress in recent years yielding performant techniques which can extract text at the character, word or line level. However, extracting blocks of text from images (block-level text spotting) is relatively unexplored. Blocks contain more context than individual lines, words or characters and so block-level text spotting would enhance downstream applications, such as translation, which benefit from added context. We propose a novel method, BTS-LLM (Block-level Text Spotting with LLMs), to identify text at the block level. BTS-LLM has three parts: 1) detecting and recognizing text at the line level, 2) grouping lines into blocks and 3) finding the best order of lines within a block using a large language model (LLM). We aim to exploit the strong semantic knowledge in LLMs for accurate block-level text spotting. Consequently if the text spotted is semantically meaningful but has been corrupted during text recognition, the LLM is also able to rectify mistakes in the text and produce a reconstruction of it.