# Syntax-Aware Network for Handwritten Mathematical Expression Recognition

arXiv: https://arxiv.org/abs/2203.01601 · HTML: https://mlsanity.com/paper/2203.01601/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 47 references that are in the map.
- First posted on arXiv 3 Mar 2022.

## Builds on (most cited first, 12 of 12)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [Robust Scene Text Recognition with Automatic Rectification](https://mlsanity.com/paper/1603.03915/) (2016)
- [Image-to-Markup Generation with Coarse-to-Fine Attention](https://mlsanity.com/paper/1609.04938/) (2016)
- [Bag of Freebies for Training Object Detection Neural Networks](https://mlsanity.com/paper/1902.04103/) (2019)
- [Symmetry-constrained Rectification Network for Scene Text Recognition](https://mlsanity.com/paper/1908.01957/) (2019)
- [Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/1801.03530/) (2018)
- [Handwritten Mathematical Expression Recognition with Bidirectionally Trained Transformer](https://mlsanity.com/paper/2105.02412/) (2021)
- [Improving Attention-Based Handwritten Mathematical Expression Recognition with Scale Augmentation and Drop Attention](https://mlsanity.com/paper/2007.10092/) (2020)
- [Pattern Generation Strategies for Improving Recognition of Handwritten Mathematical Expressions](https://mlsanity.com/paper/1901.06763/) (2019)

## Built on by (most cited first, 30 of 68)

- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://mlsanity.com/paper/2412.10302/) (2024)
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://mlsanity.com/paper/2305.07895/) (2023)
- [OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning](https://mlsanity.com/paper/2501.00321/) (2024)
- [NVLM: Open Frontier-Class Multimodal LLMs](https://mlsanity.com/paper/2409.11402/) (2024)
- [Baichuan-Omni-1.5 Technical Report](https://mlsanity.com/paper/2501.15368/) (2025)
- [MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm](https://mlsanity.com/paper/2506.05218/) (2025)
- [FineVision: Open Data Is All You Need](https://mlsanity.com/paper/2510.17269/) (2025)
- [Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation](https://mlsanity.com/paper/2310.16809/) (2023)
- [FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities](https://mlsanity.com/paper/2505.20147/) (2025)
- [Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction](https://mlsanity.com/paper/2410.21169/) (2024)
- [UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition](https://mlsanity.com/paper/2404.15254/) (2024)
- [Image Over Text: Transforming Formula Recognition Evaluation with Character Detection Matching](https://mlsanity.com/paper/2409.03643/) (2024)
- [When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2207.11463/) (2022)
- [TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokens](https://mlsanity.com/paper/2410.05261/) (2024)
- [STEP3-VL-10B Technical Report](https://mlsanity.com/paper/2601.09668/) (2026)
- [SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understanding](https://mlsanity.com/paper/2503.18943/) (2025)
- [PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models](https://mlsanity.com/paper/2412.09613/) (2024)
- [MathWriting: A Dataset For Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2404.10690/) (2024)
- [ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models](https://mlsanity.com/paper/2412.07012/) (2024)
- [Valley2: Exploring Multimodal Models with Scalable Vision-Language Design](https://mlsanity.com/paper/2501.05901/) (2025)
- [M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance](https://mlsanity.com/paper/2502.18778/) (2025)
- [PosFormer: Recognizing Complex Handwritten Mathematical Expression with Position Forest Transformer](https://mlsanity.com/paper/2407.07764/) (2024)
- [POINTS: Improving Your Vision-language Model with Affordable Strategies](https://mlsanity.com/paper/2409.04828/) (2024)
- [DM0: An Embodied-Native Vision-Language-Action Model towards Physical AI](https://mlsanity.com/paper/2602.14974/) (2026)
- [DOGR: Towards Versatile Visual Document Grounding and Referring](https://mlsanity.com/paper/2411.17125/) (2024)
- [HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding](https://mlsanity.com/paper/2412.16158/) (2024)
- [TAMER: Tree-Aware Transformer for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2408.08578/) (2024)
- [Reading or Reasoning? Format Decoupled Reinforcement Learning for Document OCR](https://mlsanity.com/paper/2601.08834/) (2025)
- [PP-FormulaNet: Bridging Accuracy and Efficiency in Advanced Formula Recognition](https://mlsanity.com/paper/2503.18382/) (2025)

## Abstract (arXiv)

Handwritten mathematical expression recognition (HMER) is a challenging task that has many potential applications. Recent methods for HMER have achieved outstanding performance with an encoder-decoder architecture. However, these methods adhere to the paradigm that the prediction is made "from one character to another", which inevitably yields prediction errors due to the complicated structures of mathematical expressions or crabbed handwritings. In this paper, we propose a simple and efficient method for HMER, which is the first to incorporate syntax information into an encoder-decoder network. Specifically, we present a set of grammar rules for converting the LaTeX markup sequence of each expression into a parsing tree; then, we model the markup sequence prediction as a tree traverse process with a deep neural network. In this way, the proposed method can effectively describe the syntax context of expressions, alleviating the structure prediction errors of HMER. Experiments on three benchmark datasets demonstrate that our method achieves better recognition performance than prior arts. To further validate the effectiveness of our method, we create a large-scale dataset consisting of 100k handwritten mathematical expression images acquired from ten thousand writers. The source code, new dataset, and pre-trained models of this work will be publicly available.