# LATTE: Improving Latex Recognition for Tables and Formulae with Iterative Refinement

arXiv: https://arxiv.org/abs/2409.14201 · HTML: https://mlsanity.com/paper/2409.14201/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 21 Sep 2024.

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [Training Language Models with Language Feedback at Scale](https://mlsanity.com/paper/2303.16755/) (2023)
- [Image-to-Markup Generation with Coarse-to-Fine Attention](https://mlsanity.com/paper/1609.04938/) (2016)
- [Improving Code Generation by Training with Natural Language Feedback](https://mlsanity.com/paper/2303.16749/) (2023)
- [LeDex: Training LLMs to Better Self-Debug and Explain Code](https://mlsanity.com/paper/2405.18649/) (2024)
- [A Deep Dive into Large Language Models for Automated Bug Localization and Repair](https://mlsanity.com/paper/2404.11595/) (2024)
- [Current Status and Performance Analysis of Table Recognition in Document Images with Deep Neural Networks](https://mlsanity.com/paper/2104.14272/) (2021)
- [ConvMath: A Convolutional Sequence Network for Mathematical Expression Recognition](https://mlsanity.com/paper/2012.12619/) (2020)
- [Tables to LaTeX: structure and content extraction from scientific tables](https://mlsanity.com/paper/2210.17246/) (2022)

## Built on by (most cited first, 8 of 8)

- [Table2LaTeX-RL: High-Fidelity LaTeX Code Generation from Table Images via Reinforced Multimodal Language Models](https://mlsanity.com/paper/2509.17589/) (2025)
- [Uni-MuMER: Unified Multi-Task Fine-Tuning of Vision-Language Model for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2505.23566/) (2025)
- [CSPO: Alleviating Reward Ambiguity for Structured Table-to-LaTeX Generation](https://mlsanity.com/paper/2604.10918/) (2026)
- [TexOCR: Advancing Document OCR Models for Compilable Page-to-LaTeX Reconstruction](https://mlsanity.com/paper/2604.22880/) (2026)
- [PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documents](https://mlsanity.com/paper/2605.10341/) (2026)
- [OCR-EDR: Rendering-Aware Diagnosis and Repair for Closed-Loop OCR Improvement](https://mlsanity.com/paper/2609.03445/) (2026)
- [A2R2: Advancing Img2LaTeX Conversion via Visual Reasoning with Attention-Guided Refinement](https://mlsanity.com/paper/2507.20890/) (2025)
- [Code as Representation: A Compilable Parsing Paradigm for Academic Documents](https://mlsanity.com/paper/2608.17550/) (2026)

## Abstract (arXiv)

Portable Document Format (PDF) files are dominantly used for storing and disseminating scientific research, legal documents, and tax information. LaTeX is a popular application for creating PDF documents. Despite its advantages, LaTeX is not WYSWYG -- what you see is what you get, i.e., the LaTeX source and rendered PDF images look drastically different, especially for formulae and tables. This gap makes it hard to modify or export LaTeX sources for formulae and tables from PDF images, and existing work is still limited. First, prior work generates LaTeX sources in a single iteration and struggles with complex LaTeX formulae. Second, existing work mainly recognizes and extracts LaTeX sources for formulae; and is incapable or ineffective for tables. This paper proposes LATTE, the first iterative refinement framework for LaTeX recognition. Specifically, we propose delta-view as feedback, which compares and pinpoints the differences between a pair of rendered images of the extracted LaTeX source and the expected correct image. Such delta-view feedback enables our fault localization model to localize the faulty parts of the incorrect recognition more accurately and enables our LaTeX refinement model to repair the incorrect extraction more accurately. LATTE improves the LaTeX source extraction accuracy of both LaTeX formulae and tables, outperforming existing techniques as well as GPT-4V by at least 7.03% of exact match, with a success refinement rate of 46.08% (formula) and 25.51% (table).