# Handwritten Mathematical Expression Recognition with Bidirectionally Trained Transformer

arXiv: https://arxiv.org/abs/2105.02412 · HTML: https://mlsanity.com/paper/2105.02412/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 50 references that are in the map.
- First posted on arXiv 6 May 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [Focusing Attention: Towards Accurate Text Recognition in Natural Images](https://mlsanity.com/paper/1709.02054/) (2017)
- [Pay Attention to What You Read: Non-recurrent Handwritten Text-Line Recognition](https://mlsanity.com/paper/2005.13044/) (2020)
- [Regularizing Neural Machine Translation by Target-bidirectional Agreement](https://mlsanity.com/paper/1808.04064/) (2018)
- [Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/1801.03530/) (2018)
- [Synchronous Bidirectional Neural Machine Translation](https://mlsanity.com/paper/1905.04847/) (2019)
- [Automatic Generation of Headlines for Online Math Questions](https://mlsanity.com/paper/1912.00839/) (2019)
- [ConvMath: A Convolutional Sequence Network for Mathematical Expression Recognition](https://mlsanity.com/paper/2012.12619/) (2020)
- [Mathematics Content Understanding for Cyberlearning via Formula Evolution Map](https://mlsanity.com/paper/1812.11786/) (2018)

## Built on by (most cited first, 30 of 34)

- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [Syntax-Aware Network for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2203.01601/) (2022)
- [DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognition](https://mlsanity.com/paper/2203.12273/) (2022)
- [UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition](https://mlsanity.com/paper/2404.15254/) (2024)
- [Image Over Text: Transforming Formula Recognition Evaluation with Character Detection Matching](https://mlsanity.com/paper/2409.03643/) (2024)
- [When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2207.11463/) (2022)
- [CoMER: Modeling Coverage for Transformer-based Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2207.04410/) (2022)
- [Handwritten Mathematical Expression Recognition via Attention Aggregation based Bi-directional Mutual Learning](https://mlsanity.com/paper/2112.03603/) (2021)
- [PosFormer: Recognizing Complex Handwritten Mathematical Expression with Position Forest Transformer](https://mlsanity.com/paper/2407.07764/) (2024)
- [TAMER: Tree-Aware Transformer for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2408.08578/) (2024)
- [PP-FormulaNet: Bridging Accuracy and Efficiency in Advanced Formula Recognition](https://mlsanity.com/paper/2503.18382/) (2025)
- [Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training](https://mlsanity.com/paper/2603.23885/) (2026)
- [Multi-Cell Decoder and Mutual Learning for Table Structure and Character Recognition](https://mlsanity.com/paper/2404.13268/) (2024)
- [Semantic Graph Representation Learning for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2308.10493/) (2023)
- [ICAL: Implicit Character-Aided Learning for Enhanced Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2405.09032/) (2024)
- [Uni-MuMER: Unified Multi-Task Fine-Tuning of Vision-Language Model for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2505.23566/) (2025)
- [Local and Global Graph Modeling with Edge-weighted Graph Attention Network for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2410.18555/) (2024)
- [Offline Handwritten Mathematical Recognition using Adversarial Learning and Transformers](https://mlsanity.com/paper/2208.09662/) (2022)
- [Bidirectional Trained Tree-Structured Decoder for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2401.00435/) (2023)
- [PEaCE: A Chemistry-Oriented Dataset for Optical Character Recognition on Scientific Documents](https://mlsanity.com/paper/2403.15724/) (2024)
- [RFL: Simplifying Chemical Structure Recognition with Ring-Free Language](https://mlsanity.com/paper/2412.07594/) (2024)
- [MMHMER:Multi-viewer and Multi-task for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2502.05557/) (2025)
- [MFH: Marrying Frequency Domain with Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2507.00430/) (2025)
- [The Return of Structural Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2508.19773/) (2025)
- [GryphOne: Symbol-Aware Masked Diffusion for Structural Refinement in Offline Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2602.03370/) (2026)
- [DenseBAM-GI: Attention Augmented DeneseNet with momentum aided GRU for HMER](https://mlsanity.com/paper/2306.16482/) (2023)
- [Attention Guidance Mechanism for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2403.01756/) (2024)
- [Mask & Match: Learning to Recognize Handwritten Math with Self-Supervised Attention](https://mlsanity.com/paper/2508.06107/) (2025)
- [Link prediction Graph Neural Networks for structure recognition of Handwritten Mathematical Expressions](https://mlsanity.com/paper/2511.02288/) (2025)
- [Hierarchical Modeling Approach to Fast and Accurate Table Recognition](https://mlsanity.com/paper/2512.21083/) (2025)

## Abstract (arXiv)

Encoder-decoder models have made great progress on handwritten mathematical expression recognition recently. However, it is still a challenge for existing methods to assign attention to image features accurately. Moreover, those encoder-decoder models usually adopt RNN-based models in their decoder part, which makes them inefficient in processing long LaTeX sequences. In this paper, a transformer-based decoder is employed to replace RNN-based ones, which makes the whole model architecture very concise. Furthermore, a novel training strategy is introduced to fully exploit the potential of the transformer in bidirectional language modeling. Compared to several methods that do not use data augmentation, experiments demonstrate that our model improves the ExpRate of current state-of-the-art methods on CROHME 2014 by 2.23%. Similarly, on CROHME 2016 and CROHME 2019, we improve the ExpRate by 1.92% and 2.28% respectively.