# ChartDETR: A Multi-shape Detection Network for Visual Chart Recognition

arXiv: https://arxiv.org/abs/2308.07743 · HTML: https://mlsanity.com/paper/2308.07743/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 15 Aug 2023.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation](https://mlsanity.com/paper/1908.10357/) (2019)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [DePlot: One-shot visual language reasoning by plot-to-table translation](https://mlsanity.com/paper/2212.10505/) (2022)
- [DETRs with Hybrid Matching](https://mlsanity.com/paper/2207.13080/) (2022)
- [PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model](https://mlsanity.com/paper/1803.08225/) (2018)
- [PifPaf: Composite Fields for Human Pose Estimation](https://mlsanity.com/paper/1903.06593/) (2019)
- [Data Extraction from Charts via Single Deep Neural Network](https://mlsanity.com/paper/1906.11906/) (2019)
- [Chart-Text: A Fully Automated Chart Image Descriptor](https://mlsanity.com/paper/1812.10636/) (2018)
- [CHARTER: heatmap-based multi-type chart data extraction](https://mlsanity.com/paper/2111.14103/) (2021)

## Built on by (most cited first, 7 of 7)

- [SIMPLOT: Enhancing Chart Question Answering by Distilling Essentials](https://mlsanity.com/paper/2405.00021/) (2024)
- [Advancing Chart Question Answering with Robust Chart Component Recognition](https://mlsanity.com/paper/2407.21038/) (2024)
- [Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancement](https://mlsanity.com/paper/2602.10138/) (2026)
- [Evaluating Graphical Perception Capabilities of Vision Transformers](https://mlsanity.com/paper/2602.18178/) (2026)
- [Do Large Language Models Understand Data Visualization Principles?](https://mlsanity.com/paper/2602.20084/) (2026)
- [De-rendering, Reasoning, and Repairing Charts with Vision-Language Models](https://mlsanity.com/paper/2602.20291/) (2026)
- [Bar-JEPA: Extracting Values from Bar Chart with Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2608.06062/) (2026)

## Abstract (arXiv)

Visual chart recognition systems are gaining increasing attention due to the growing demand for automatically identifying table headers and values from chart images. Current methods rely on keypoint detection to estimate data element shapes in charts but suffer from grouping errors in post-processing. To address this issue, we propose ChartDETR, a transformer-based multi-shape detector that localizes keypoints at the corners of regular shapes to reconstruct multiple data elements in a single chart image. Our method predicts all data element shapes at once by introducing query groups in set prediction, eliminating the need for further postprocessing. This property allows ChartDETR to serve as a unified framework capable of representing various chart types without altering the network architecture, effectively detecting data elements of diverse shapes. We evaluated ChartDETR on three datasets, achieving competitive results across all chart types without any additional enhancements. For example, ChartDETR achieved an F1 score of 0.98 on Adobe Synthetic, significantly outperforming the previous best model with a 0.71 F1 score. Additionally, we obtained a new state-of-the-art result of 0.97 on ExcelChart400k. The code will be made publicly available.