# Rethinking the Pointer Loss in Table Structure Recognition: Geometry-Aware Pointer Loss for Spatial Locality

arXiv: https://arxiv.org/abs/2606.18721 · HTML: https://mlsanity.com/paper/2606.18721/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Builds on (most cited first, 16 of 16)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [Image-based table recognition: data, model, and evaluation](https://mlsanity.com/paper/1911.10683/) (2019)
- [TableFormer: Table Structure Understanding with Transformers](https://mlsanity.com/paper/2203.01017/) (2022)
- [Complicated Table Structure Recognition](https://mlsanity.com/paper/1908.04729/) (2019)
- [Split, embed and merge: An accurate table structure recognizer](https://mlsanity.com/paper/2107.05214/) (2021)
- [Improving Table Structure Recognition with Visual-Alignment Sequential Coordinate Modeling](https://mlsanity.com/paper/2303.06949/) (2023)
- [PingAn-VCGroup's Solution for ICDAR 2021 Competition on Scientific Literature Parsing Task B: Table Recognition to HTML](https://mlsanity.com/paper/2105.01848/) (2021)
- [Optimized Table Tokenization for Table Structure Recognition](https://mlsanity.com/paper/2305.03393/) (2023)
- [Robust Table Detection and Structure Recognition from Heterogeneous Document Images](https://mlsanity.com/paper/2203.09056/) (2022)
- [GriTS: Grid table similarity metric for table structure recognition](https://mlsanity.com/paper/2203.12555/) (2022)
- [TGRNet: A Table Graph Reconstruction Network for Table Structure Recognition](https://mlsanity.com/paper/2106.10598/) (2021)
- [Robust Table Structure Recognition with Dynamic Queries Enhanced Detection Transformer](https://mlsanity.com/paper/2303.11615/) (2023)
- [TFLOP: Table Structure Recognition Framework with Layout Pointer Mechanism](https://mlsanity.com/paper/2501.11800/) (2025)
- [SPRINT: Script-agnostic Structure Recognition in Tables](https://mlsanity.com/paper/2503.11932/) (2025)

## Abstract (arXiv)

Table Structure Recognition (TSR) using a pointer network achieves impressive results by predicting HTML sequences while aligning tags to detected text (or cell) regions. However, our analysis reveals that when pointer networks fail, 79.6% of errors occur between spatially adjacent cells (Manhattan distance <= 2). Despite this, standard cross-entropy loss weights all negative candidates equally. In this work, we propose Geometry-Aware Pointer (GAP) Loss, which reweights the cross-entropy objective based on spatial proximity to ground truth. By applying inverse distance weighting, GAP focuses gradient flow where the model struggles most: immediate neighbors receive stronger gradients than distant cells. Our approach requires only a straightforward modification to the loss computation, maintaining the same model architecture with zero additional inference cost. Extensive experiments on PubTabNet and SynthTabNet demonstrate that GAP consistently reduces adjacent-cell errors, achieving new state-of-the-art performance. Our findings suggest that incorporating geometric inductive biases at the loss level provides a simple yet effective approach to robust TSR. Our code is available at https://github.com/teamreboott/GAP