# Rethinking Detection Based Table Structure Recognition for Visually Rich Document Images

arXiv: https://arxiv.org/abs/2312.00699 · HTML: https://mlsanity.com/paper/2312.00699/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 65 references that are in the map.
- First posted on arXiv 1 Dec 2023.

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation](https://mlsanity.com/paper/2209.08575/) (2022)
- [Image-based table recognition: data, model, and evaluation](https://mlsanity.com/paper/1911.10683/) (2019)
- [Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs](https://mlsanity.com/paper/2203.06717/) (2022)
- [Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context](https://mlsanity.com/paper/2005.00589/) (2020)
- [PubTables-1M: Towards comprehensive table extraction from unstructured documents](https://mlsanity.com/paper/2110.00061/) (2021)
- [CascadeTabNet: An approach for end to end table detection and structure recognition from image-based documents](https://mlsanity.com/paper/2004.12629/) (2020)
- [TableFormer: Table Structure Understanding with Transformers](https://mlsanity.com/paper/2203.01017/) (2022)
- [Complicated Table Structure Recognition](https://mlsanity.com/paper/1908.04729/) (2019)
- [Dense Distinct Query for End-to-End Object Detection](https://mlsanity.com/paper/2303.12776/) (2023)
- [What Makes for End-to-End Object Detection?](https://mlsanity.com/paper/2012.05780/) (2020)

## Built on by (most cited first, 1 of 1)

- [SPRINT: Script-agnostic Structure Recognition in Tables](https://mlsanity.com/paper/2503.11932/) (2025)

## Abstract (arXiv)

Table Structure Recognition (TSR) is a widely discussed task aiming at transforming unstructured table images into structured formats, such as HTML sequences, to make text-only models, such as ChatGPT, that can further process these tables. One type of solution is using detection models to detect table components, such as columns and rows, then applying a rule-based post-processing method to convert detection results into HTML sequences. However, existing detection-based models usually cannot perform as well as other types of solutions regarding cell-level TSR metrics, such as TEDS, and the underlying reasons limiting the performance of these models on the TSR task are also not well-explored. Therefore, we revisit existing detection-based models comprehensively and explore the underlying reasons hindering these models' performance, including the improper problem definition, the mismatch issue of detection and TSR metrics, the characteristics of detection models, and the impact of local and long-range features extraction. Based on our analysis and findings, we apply simple methods to tailor a typical two-stage detection model, Cascade R-CNN, for the TSR task. The experimental results show that the tailored Cascade R-CNN based model can improve the base Cascade R-CNN model by 16.35% on the FinTabNet dataset regarding the structure-only TEDS, outperforming other types of state-of-the-art methods, demonstrating that our findings can be a guideline for improving detection-based TSR models and that a purely detection-based solution is competitive with other types of solutions, such as graph-based and image-to-sequence solutions.