# When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition

arXiv: https://arxiv.org/abs/2207.11463 · HTML: https://mlsanity.com/paper/2207.11463/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 46 references that are in the map.
- First posted on arXiv 23 Jul 2022.

## Builds on (most cited first, 20 of 20)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes](https://mlsanity.com/paper/1802.10062/) (2018)
- [Modeling Coverage for Neural Machine Translation](https://mlsanity.com/paper/1601.04811/) (2016)
- [Image-to-Markup Generation with Coarse-to-Fine Attention](https://mlsanity.com/paper/1609.04938/) (2016)
- [Syntax-Aware Network for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2203.01601/) (2022)
- [Where are the Blobs: Counting by Localization with Point Supervision](https://mlsanity.com/paper/1807.09856/) (2018)
- [Perspective-Guided Convolution Networks for Crowd Counting](https://mlsanity.com/paper/1909.06966/) (2019)
- [Bag of Freebies for Training Object Detection Neural Networks](https://mlsanity.com/paper/1902.04103/) (2019)
- [Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/1801.03530/) (2018)
- [AutoScale: Learning to Scale for Crowd Counting and Localization](https://mlsanity.com/paper/1912.09632/) (2019)
- [Handwritten Mathematical Expression Recognition with Bidirectionally Trained Transformer](https://mlsanity.com/paper/2105.02412/) (2021)
- [Aggregation Cross-Entropy for Sequence Recognition](https://mlsanity.com/paper/1904.08364/) (2019)
- [Improving Attention-Based Handwritten Mathematical Expression Recognition with Scale Augmentation and Drop Attention](https://mlsanity.com/paper/2007.10092/) (2020)
- [Handwritten Mathematical Expression Recognition via Attention Aggregation based Bi-directional Mutual Learning](https://mlsanity.com/paper/2112.03603/) (2021)

## Built on by (most cited first, 30 of 35)

- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://mlsanity.com/paper/2305.07895/) (2023)
- [Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation](https://mlsanity.com/paper/2310.16809/) (2023)
- [Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction](https://mlsanity.com/paper/2410.21169/) (2024)
- [UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition](https://mlsanity.com/paper/2404.15254/) (2024)
- [Image Over Text: Transforming Formula Recognition Evaluation with Character Detection Matching](https://mlsanity.com/paper/2409.03643/) (2024)
- [Context Perception Parallel Decoder for Scene Text Recognition](https://mlsanity.com/paper/2307.12270/) (2023)
- [PosFormer: Recognizing Complex Handwritten Mathematical Expression with Position Forest Transformer](https://mlsanity.com/paper/2407.07764/) (2024)
- [UPOCR: Towards Unified Pixel-Level OCR Interface](https://mlsanity.com/paper/2312.02694/) (2023)
- [CoHD: A Counting-Aware Hierarchical Decoding Framework for Generalized Referring Expression Segmentation](https://mlsanity.com/paper/2405.15658/) (2024)
- [TAMER: Tree-Aware Transformer for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2408.08578/) (2024)
- [PP-FormulaNet: Bridging Accuracy and Efficiency in Advanced Formula Recognition](https://mlsanity.com/paper/2503.18382/) (2025)
- [Masked and Permuted Implicit Context Learning for Scene Text Recognition](https://mlsanity.com/paper/2305.16172/) (2023)
- [Semantic Graph Representation Learning for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2308.10493/) (2023)
- [MathNet: A Data-Centric Approach for Printed Mathematical Expression Recognition](https://mlsanity.com/paper/2404.13667/) (2024)
- [ICAL: Implicit Character-Aided Learning for Enhanced Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2405.09032/) (2024)
- [Infinity-Parser2 Technical Report](https://mlsanity.com/paper/2607.07836/) (2026)
- [Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding](https://mlsanity.com/paper/2505.05446/) (2025)
- [Uni-MuMER: Unified Multi-Task Fine-Tuning of Vision-Language Model for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2505.23566/) (2025)
- [DocTron-Formula: Generalized Formula Recognition in Complex and Structured Scenarios](https://mlsanity.com/paper/2508.00311/) (2025)
- [DREAM: Document Reconstruction via End-to-end Autoregressive Model](https://mlsanity.com/paper/2507.05805/) (2025)
- [Count, Decode and Fetch: A New Approach to Handwritten Chinese Character Error Correction](https://mlsanity.com/paper/2307.16253/) (2023)
- [Recognizing Handwritten Mathematical Expressions of Vertical Addition and Subtraction](https://mlsanity.com/paper/2308.05820/) (2023)
- [Bidirectional Trained Tree-Structured Decoder for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2401.00435/) (2023)
- [Enhancing Complex Formula Recognition with Hierarchical Detail-Focused Network](https://mlsanity.com/paper/2409.11677/) (2024)
- [MMHMER:Multi-viewer and Multi-task for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2502.05557/) (2025)
- [SemiHMER: Semi-supervised Handwritten Mathematical Expression Recognition using pseudo-labels](https://mlsanity.com/paper/2502.07172/) (2025)
- [MFH: Marrying Frequency Domain with Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2507.00430/) (2025)
- [GryphOne: Symbol-Aware Masked Diffusion for Structural Refinement in Offline Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2602.03370/) (2026)
- [An Intelligent-Detection Network for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2311.15273/) (2023)
- [Attention Guidance Mechanism for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2403.01756/) (2024)

## Abstract (arXiv)

Recently, most handwritten mathematical expression recognition (HMER) methods adopt the encoder-decoder networks, which directly predict the markup sequences from formula images with the attention mechanism. However, such methods may fail to accurately read formulas with complicated structure or generate long markup sequences, as the attention results are often inaccurate due to the large variance of writing styles or spatial layouts. To alleviate this problem, we propose an unconventional network for HMER named Counting-Aware Network (CAN), which jointly optimizes two tasks: HMER and symbol counting. Specifically, we design a weakly-supervised counting module that can predict the number of each symbol class without the symbol-level position annotations, and then plug it into a typical attention-based encoder-decoder model for HMER. Experiments on the benchmark datasets for HMER validate that both joint optimization and counting results are beneficial for correcting the prediction errors of encoder-decoder models, and CAN consistently outperforms the state-of-the-art methods. In particular, compared with an encoder-decoder model for HMER, the extra time cost caused by the proposed counting module is marginal. The source code is available at https://github.com/LBH1024/CAN.