# PosFormer: Recognizing Complex Handwritten Mathematical Expression with Position Forest Transformer

arXiv: https://arxiv.org/abs/2407.07764 · HTML: https://mlsanity.com/paper/2407.07764/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 47 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition](https://mlsanity.com/paper/2007.07542/) (2020)
- [Image-to-Markup Generation with Coarse-to-Fine Attention](https://mlsanity.com/paper/1609.04938/) (2016)
- [Syntax-Aware Network for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2203.01601/) (2022)
- [UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition](https://mlsanity.com/paper/2404.15254/) (2024)
- [Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/1801.03530/) (2018)
- [When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2207.11463/) (2022)
- [Handwritten Mathematical Expression Recognition with Bidirectionally Trained Transformer](https://mlsanity.com/paper/2105.02412/) (2021)
- [Self-supervised Character-to-Character Distillation for Text Recognition](https://mlsanity.com/paper/2211.00288/) (2022)
- [Self-supervised Implicit Glyph Attention for Text Recognition](https://mlsanity.com/paper/2203.03382/) (2022)
- [CoMER: Modeling Coverage for Transformer-based Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2207.04410/) (2022)
- [Improving Attention-Based Handwritten Mathematical Expression Recognition with Scale Augmentation and Drop Attention](https://mlsanity.com/paper/2007.10092/) (2020)
- [Handwritten Mathematical Expression Recognition via Attention Aggregation based Bi-directional Mutual Learning](https://mlsanity.com/paper/2112.03603/) (2021)
- [Industrial Scene Text Detection with Refined Feature-attentive Network](https://mlsanity.com/paper/2110.12663/) (2021)
- [Pattern Generation Strategies for Improving Recognition of Handwritten Mathematical Expressions](https://mlsanity.com/paper/1901.06763/) (2019)
- [Bridging Synthetic and Real Worlds for Pre-training Scene Text Detectors](https://mlsanity.com/paper/2312.05286/) (2023)

## Built on by (most cited first, 18 of 18)

- [Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understanding](https://mlsanity.com/paper/2503.14140/) (2025)
- [A Token-level Text Image Foundation Model for Document Understanding](https://mlsanity.com/paper/2503.02304/) (2025)
- [Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review](https://mlsanity.com/paper/2502.16586/) (2025)
- [Uni-MuMER: Unified Multi-Task Fine-Tuning of Vision-Language Model for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2505.23566/) (2025)
- [DocTron-Formula: Generalized Formula Recognition in Complex and Structured Scenarios](https://mlsanity.com/paper/2508.00311/) (2025)
- [CodePercept: Code-Grounded Visual STEM Perception for MLLMs](https://mlsanity.com/paper/2603.10757/) (2026)
- [Local and Global Graph Modeling with Edge-weighted Graph Attention Network for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2410.18555/) (2024)
- [MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosis](https://mlsanity.com/paper/2508.08275/) (2025)
- [MMHMER:Multi-viewer and Multi-task for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2502.05557/) (2025)
- [SemiHMER: Semi-supervised Handwritten Mathematical Expression Recognition using pseudo-labels](https://mlsanity.com/paper/2502.07172/) (2025)
- [Simple Vision-Language Math Reasoning via Rendered Text](https://mlsanity.com/paper/2511.11704/) (2025)
- [GryphOne: Symbol-Aware Masked Diffusion for Structural Refinement in Offline Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2602.03370/) (2026)
- [LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinement](https://mlsanity.com/paper/2606.23539/) (2026)
- [ViTexQA: A Multi-Frame Temporal Perception Dataset for Video Text Question Answering](https://mlsanity.com/paper/2606.24602/) (2026)
- [Towards Scalable Training for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2508.09220/) (2025)
- [DiffMath: Symbol- and Graph-Aware Latent Diffusion Transformer for Handwritten Mathematical Expression Generation](https://mlsanity.com/paper/2606.19939/) (2026)
- [PA-CDM: Position-Aware Character Detection Matching for Evaluating Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2609.12917/) (2026)
- [Cross-Domain Few-Shot Writer Adaptation for Real-World Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2609.32513/) (2026)

## Abstract (arXiv)

Handwritten Mathematical Expression Recognition (HMER) has wide applications in human-machine interaction scenarios, such as digitized education and automated offices. Recently, sequence-based models with encoder-decoder architectures have been commonly adopted to address this task by directly predicting LaTeX sequences of expression images. However, these methods only implicitly learn the syntax rules provided by LaTeX, which may fail to describe the position and hierarchical relationship between symbols due to complex structural relations and diverse handwriting styles. To overcome this challenge, we propose a position forest transformer (PosFormer) for HMER, which jointly optimizes two tasks: expression recognition and position recognition, to explicitly enable position-aware symbol feature representation learning. Specifically, we first design a position forest that models the mathematical expression as a forest structure and parses the relative position relationships between symbols. Without requiring extra annotations, each symbol is assigned a position identifier in the forest to denote its relative spatial position. Second, we propose an implicit attention correction module to accurately capture attention for HMER in the sequence-based decoder architecture. Extensive experiments validate the superiority of PosFormer, which consistently outperforms the state-of-the-art methods 2.03%/1.22%/2.00%, 1.83%, and 4.62% gains on the single-line CROHME 2014/2016/2019, multi-line M2E, and complex MNE datasets, respectively, with no additional latency or computational cost. Code is available at https://github.com/SJTU-DeepVisionLab/PosFormer.