# Handwritten Mathematical Expression Recognition via Attention Aggregation based Bi-directional Mutual Learning

arXiv: https://arxiv.org/abs/2112.03603 · HTML: https://mlsanity.com/paper/2112.03603/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 7 Dec 2021.

## Builds on (most cited first, 22 of 22)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Knowledge Distillation by On-the-Fly Native Ensemble](https://mlsanity.com/paper/1806.04606/) (2018)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [Large scale distributed neural network training through online distillation](https://mlsanity.com/paper/1804.03235/) (2018)
- [Image-to-Markup Generation with Coarse-to-Fine Attention](https://mlsanity.com/paper/1609.04938/) (2016)
- [Collaborative Learning for Deep Neural Networks](https://mlsanity.com/paper/1805.11761/) (2018)
- [Motion-Attentive Transition for Zero-Shot Video Object Segmentation](https://mlsanity.com/paper/2003.04253/) (2020)
- [Addressing the Rare Word Problem in Neural Machine Translation](https://mlsanity.com/paper/1410.8206/) (2014)
- [Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/1801.03530/) (2018)
- [Handwritten Mathematical Expression Recognition with Bidirectionally Trained Transformer](https://mlsanity.com/paper/2105.02412/) (2021)
- [Improving Attention-Based Handwritten Mathematical Expression Recognition with Scale Augmentation and Drop Attention](https://mlsanity.com/paper/2007.10092/) (2020)
- [Pattern Generation Strategies for Improving Recognition of Handwritten Mathematical Expressions](https://mlsanity.com/paper/1901.06763/) (2019)
- [ConvMath: A Convolutional Sequence Network for Mathematical Expression Recognition](https://mlsanity.com/paper/2012.12619/) (2020)
- [A Bayesian model for recognizing handwritten mathematical expressions](https://mlsanity.com/paper/1409.5317/) (2014)

## Built on by (most cited first, 20 of 20)

- [Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation](https://mlsanity.com/paper/2310.16809/) (2023)
- [UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition](https://mlsanity.com/paper/2404.15254/) (2024)
- [Image Over Text: Transforming Formula Recognition Evaluation with Character Detection Matching](https://mlsanity.com/paper/2409.03643/) (2024)
- [When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2207.11463/) (2022)
- [PosFormer: Recognizing Complex Handwritten Mathematical Expression with Position Forest Transformer](https://mlsanity.com/paper/2407.07764/) (2024)
- [TAMER: Tree-Aware Transformer for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2408.08578/) (2024)
- [DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generation](https://mlsanity.com/paper/2410.13726/) (2024)
- [PP-FormulaNet: Bridging Accuracy and Efficiency in Advanced Formula Recognition](https://mlsanity.com/paper/2503.18382/) (2025)
- [Semantic Graph Representation Learning for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2308.10493/) (2023)
- [MathNet: A Data-Centric Approach for Printed Mathematical Expression Recognition](https://mlsanity.com/paper/2404.13667/) (2024)
- [ICAL: Implicit Character-Aided Learning for Enhanced Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2405.09032/) (2024)
- [Uni-MuMER: Unified Multi-Task Fine-Tuning of Vision-Language Model for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2505.23566/) (2025)
- [Recognizing Handwritten Mathematical Expressions of Vertical Addition and Subtraction](https://mlsanity.com/paper/2308.05820/) (2023)
- [Bidirectional Trained Tree-Structured Decoder for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2401.00435/) (2023)
- [Enhancing Complex Formula Recognition with Hierarchical Detail-Focused Network](https://mlsanity.com/paper/2409.11677/) (2024)
- [SemiHMER: Semi-supervised Handwritten Mathematical Expression Recognition using pseudo-labels](https://mlsanity.com/paper/2502.07172/) (2025)
- [MFH: Marrying Frequency Domain with Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2507.00430/) (2025)
- [Attention Guidance Mechanism for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2403.01756/) (2024)
- [Towards Scalable Training for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2508.09220/) (2025)
- [Texo: Formula Recognition within 20M Parameters](https://mlsanity.com/paper/2602.17189/) (2026)

## Abstract (arXiv)

Handwritten mathematical expression recognition aims to automatically generate LaTeX sequences from given images. Currently, attention-based encoder-decoder models are widely used in this task. They typically generate target sequences in a left-to-right (L2R) manner, leaving the right-to-left (R2L) contexts unexploited. In this paper, we propose an Attention aggregation based Bi-directional Mutual learning Network (ABM) which consists of one shared encoder and two parallel inverse decoders (L2R and R2L). The two decoders are enhanced via mutual distillation, which involves one-to-one knowledge transfer at each training step, making full use of the complementary information from two inverse directions. Moreover, in order to deal with mathematical symbols in diverse scales, an Attention Aggregation Module (AAM) is proposed to effectively integrate multi-scale coverage attentions. Notably, in the inference phase, given that the model already learns knowledge from two inverse directions, we only use the L2R branch for inference, keeping the original parameter size and inference speed. Extensive experiments demonstrate that our proposed approach achieves the recognition accuracy of 56.85 % on CROHME 2014, 52.92 % on CROHME 2016, and 53.96 % on CROHME 2019 without data augmentation and model ensembling, substantially outperforming the state-of-the-art methods. The source code is available in https://github.com/XH-B/ABM.