# An End-to-End Multi-Task Learning Model for Image-based Table Recognition

arXiv: https://arxiv.org/abs/2303.08648 · HTML: https://mlsanity.com/paper/2303.08648/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 15 Mar 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Image-based table recognition: data, model, and evaluation](https://mlsanity.com/paper/1911.10683/) (2019)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context](https://mlsanity.com/paper/2005.00589/) (2020)
- [TableFormer: Table Structure Understanding with Transformers](https://mlsanity.com/paper/2203.01017/) (2022)
- [Image-to-Markup Generation with Coarse-to-Fine Attention](https://mlsanity.com/paper/1609.04938/) (2016)
- [MASTER: Multi-Aspect Non-local Network for Scene Text Recognition](https://mlsanity.com/paper/1910.02562/) (2019)
- [Table Structure Recognition using Top-Down and Bottom-Up Cues](https://mlsanity.com/paper/2010.04565/) (2020)
- [LGPMA: Complicated Table Structure Recognition with Local and Global Pyramid Mask Alignment](https://mlsanity.com/paper/2105.06224/) (2021)
- [Split, embed and merge: An accurate table structure recognizer](https://mlsanity.com/paper/2107.05214/) (2021)
- [PingAn-VCGroup's Solution for ICDAR 2021 Competition on Scientific Literature Parsing Task B: Table Recognition to HTML](https://mlsanity.com/paper/2105.01848/) (2021)
- [ICDAR 2021 Competition on Scientific Literature Parsing](https://mlsanity.com/paper/2106.14616/) (2021)
- [PingAn-VCGroup's Solution for ICDAR 2021 Competition on Scientific Table Image Recognition to Latex](https://mlsanity.com/paper/2105.01846/) (2021)
- [ICDAR 2021 Competition on Scientific Table Image Recognition to LaTeX](https://mlsanity.com/paper/2105.14426/) (2021)
- [The Benefits of Close-Domain Fine-Tuning for Table Detection in Document Images](https://mlsanity.com/paper/1912.05846/) (2019)

## Built on by (most cited first, 14 of 14)

- [PaliGemma 2: A Family of Versatile VLMs for Transfer](https://mlsanity.com/paper/2412.03555/) (2024)
- [Multi-Cell Decoder and Mutual Learning for Table Structure and Character Recognition](https://mlsanity.com/paper/2404.13268/) (2024)
- [Vision Language Models for Spreadsheet Understanding: Challenges and Opportunities](https://mlsanity.com/paper/2405.16234/) (2024)
- [CREPE: Coordinate-Aware End-to-End Document Parser](https://mlsanity.com/paper/2405.00260/) (2024)
- [Rethinking Detection Based Table Structure Recognition for Visually Rich Document Images](https://mlsanity.com/paper/2312.00699/) (2023)
- [PdfTable: A Unified Toolkit for Deep Learning-Based Table Extraction](https://mlsanity.com/paper/2409.05125/) (2024)
- [Title block detection and information extraction for enhanced building drawings search](https://mlsanity.com/paper/2504.08645/) (2025)
- [InstructTable: Improving Table Structure Recognition Through Instructions](https://mlsanity.com/paper/2604.02880/) (2026)
- [TabIQA: Table Questions Answering on Business Document Images](https://mlsanity.com/paper/2303.14935/) (2023)
- [Hierarchical Modeling Approach to Fast and Accurate Table Recognition](https://mlsanity.com/paper/2512.21083/) (2025)
- [TableSeq: Unified Generation of Structure, Content, and Layout](https://mlsanity.com/paper/2604.16070/) (2026)
- [FastTab: A Fast Table Recognizer with a Tiny Recursive Module and 1D Transformers](https://mlsanity.com/paper/2605.22422/) (2026)
- [Revisiting Structural Dependency in Autoregressive Multi-Task Table Recognition via Order-Independent Cell-Level Representations](https://mlsanity.com/paper/2606.17874/) (2026)
- [Tables Decoded: DELTA for Structure, TARQA for Understanding](https://mlsanity.com/paper/2609.17458/) (2026)

## Abstract (arXiv)

Image-based table recognition is a challenging task due to the diversity of table styles and the complexity of table structures. Most of the previous methods focus on a non-end-to-end approach which divides the problem into two separate sub-problems: table structure recognition; and cell-content recognition and then attempts to solve each sub-problem independently using two separate systems. In this paper, we propose an end-to-end multi-task learning model for image-based table recognition. The proposed model consists of one shared encoder, one shared decoder, and three separate decoders which are used for learning three sub-tasks of table recognition: table structure recognition, cell detection, and cell-content recognition. The whole system can be easily trained and inferred in an end-to-end approach. In the experiments, we evaluate the performance of the proposed model on two large-scale datasets: FinTabNet and PubTabNet. The experiment results show that the proposed model outperforms the state-of-the-art methods in all benchmark datasets.