# UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition

arXiv: https://arxiv.org/abs/2404.15254 · HTML: https://mlsanity.com/paper/2404.15254/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [OCR-free Document Understanding Transformer](https://mlsanity.com/paper/2111.15664/) (2021)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://mlsanity.com/paper/2401.16420/) (2024)
- [CMT: Convolutional Neural Networks Meet Vision Transformers](https://mlsanity.com/paper/2107.06263/) (2021)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models](https://mlsanity.com/paper/2312.06109/) (2023)
- [VIGC: Visual Instruction Generation and Correction](https://mlsanity.com/paper/2308.12714/) (2023)
- [Image-to-Markup Generation with Coarse-to-Fine Attention](https://mlsanity.com/paper/1609.04938/) (2016)
- [Syntax-Aware Network for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2203.01601/) (2022)
- [Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/1801.03530/) (2018)
- [When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2207.11463/) (2022)
- [Handwritten Mathematical Expression Recognition with Bidirectionally Trained Transformer](https://mlsanity.com/paper/2105.02412/) (2021)
- [CoMER: Modeling Coverage for Transformer-based Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2207.04410/) (2022)
- [Improving Attention-Based Handwritten Mathematical Expression Recognition with Scale Augmentation and Drop Attention](https://mlsanity.com/paper/2007.10092/) (2020)
- [Handwritten Mathematical Expression Recognition via Attention Aggregation based Bi-directional Mutual Learning](https://mlsanity.com/paper/2112.03603/) (2021)
- [Pattern Generation Strategies for Improving Recognition of Handwritten Mathematical Expressions](https://mlsanity.com/paper/1901.06763/) (2019)

## Built on by (most cited first, 30 of 42)

- [MinerU: An Open-Source Solution for Precise Document Content Extraction](https://mlsanity.com/paper/2409.18839/) (2024)
- [PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model](https://mlsanity.com/paper/2510.14528/) (2025)
- [MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm](https://mlsanity.com/paper/2506.05218/) (2025)
- [Dolphin: Document Image Parsing via Heterogeneous Anchor Prompting](https://mlsanity.com/paper/2505.14059/) (2025)
- [Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction](https://mlsanity.com/paper/2410.21169/) (2024)
- [Image Over Text: Transforming Formula Recognition Evaluation with Character Detection Matching](https://mlsanity.com/paper/2409.03643/) (2024)
- [DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models](https://mlsanity.com/paper/2406.11633/) (2024)
- [TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokens](https://mlsanity.com/paper/2410.05261/) (2024)
- [MinerU2.5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scale](https://mlsanity.com/paper/2604.04771/) (2026)
- [OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.02592/) (2024)
- [GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-training](https://mlsanity.com/paper/2412.11863/) (2024)
- [Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understanding](https://mlsanity.com/paper/2503.14140/) (2025)
- [PosFormer: Recognizing Complex Handwritten Mathematical Expression with Position Forest Transformer](https://mlsanity.com/paper/2407.07764/) (2024)
- [MonkeyOCR v1.5 Technical Report: Unlocking Robust Document Parsing for Complex Patterns](https://mlsanity.com/paper/2511.10390/) (2025)
- [UniRec-0.1B: Unified Text and Formula Recognition with 0.1B Parameters](https://mlsanity.com/paper/2512.21095/) (2025)
- [DianJin-OCR-R1: Enhancing OCR Capabilities via a Reasoning-and-Tool Interleaved Vision-Language Model](https://mlsanity.com/paper/2508.13238/) (2025)
- [MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decoding](https://mlsanity.com/paper/2603.22458/) (2026)
- [PP-FormulaNet: Bridging Accuracy and Efficiency in Advanced Formula Recognition](https://mlsanity.com/paper/2503.18382/) (2025)
- [Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training](https://mlsanity.com/paper/2603.23885/) (2026)
- [DOCR-Inspector: Fine-Grained and Automated Evaluation of Document Parsing with VLM](https://mlsanity.com/paper/2512.10619/) (2025)
- [Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence](https://mlsanity.com/paper/2606.15932/) (2026)
- [MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios](https://mlsanity.com/paper/2603.28130/) (2026)
- [Infinity-Parser2 Technical Report](https://mlsanity.com/paper/2607.07836/) (2026)
- [MonkeyOCRv2: A Visual-Text Foundation Model for Document AI](https://mlsanity.com/paper/2607.11562/) (2026)
- [Uni-MuMER: Unified Multi-Task Fine-Tuning of Vision-Language Model for Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2505.23566/) (2025)
- [DocTron-Formula: Generalized Formula Recognition in Complex and Structured Scenarios](https://mlsanity.com/paper/2508.00311/) (2025)
- [MindVL: Towards Efficient and Effective Training of Multimodal Large Language Models on Ascend NPUs](https://mlsanity.com/paper/2509.11662/) (2025)
- [Boosting Document Parsing Efficiency and Performance with Coarse-to-Fine Visual Processing](https://mlsanity.com/paper/2603.24326/) (2026)
- [HyperVL: An Efficient and Dynamic Multimodal Large Language Model for Edge Devices](https://mlsanity.com/paper/2512.14052/) (2025)
- [VisTIRA: Closing the Image-Text Modality Gap in Visual Math Reasoning via Structured Tool Integration](https://mlsanity.com/paper/2601.14440/) (2026)

## Abstract (arXiv)

The paper introduces the UniMER dataset, marking the first study on Mathematical Expression Recognition (MER) targeting complex real-world scenarios. The UniMER dataset includes a large-scale training set, UniMER-1M, which offers unprecedented scale and diversity with one million training instances to train high-quality, robust models. Additionally, UniMER features a meticulously designed, diverse test set, UniMER-Test, which covers a variety of formula distributions found in real-world scenarios, providing a more comprehensive and fair evaluation. To better utilize the UniMER dataset, the paper proposes a Universal Mathematical Expression Recognition Network (UniMERNet), tailored to the characteristics of formula recognition. UniMERNet consists of a carefully designed encoder that incorporates detail-aware and local context features, and an optimized decoder for accelerated performance. Extensive experiments conducted using the UniMER-1M dataset and UniMERNet demonstrate that training on the large-scale UniMER-1M dataset can produce a more generalizable formula recognition model, significantly outperforming all previous datasets. Furthermore, the introduction of UniMERNet enhances the model's performance in formula recognition, achieving higher accuracy and speeds. All data, models, and code are available at https://github.com/opendatalab/UniMERNet.