# MolParser: End-to-end Visual Recognition of Molecule Structures in the Wild

arXiv: https://arxiv.org/abs/2411.11098 · HTML: https://mlsanity.com/paper/2411.11098/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 71 references that are in the map.
- First posted on arXiv 17 Nov 2024.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [GraphMAE: Self-Supervised Masked Graph Autoencoders](https://mlsanity.com/paper/2205.10803/) (2022)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [Pre-training Molecular Graph Representation with 3D Geometry](https://mlsanity.com/paper/2110.07728/) (2021)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Prediction](https://mlsanity.com/paper/2106.06130/) (2021)
- [OmniParser for Pure Vision Based GUI Agent](https://mlsanity.com/paper/2408.00203/) (2024)
- [Large-Scale Representation Learning on Graphs via Bootstrapping](https://mlsanity.com/paper/2102.06514/) (2021)
- [3D Infomax improves GNNs for Molecular Property Prediction](https://mlsanity.com/paper/2110.04126/) (2021)
- [MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter](https://mlsanity.com/paper/2310.12798/) (2023)
- [Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performance](https://mlsanity.com/paper/2410.16261/) (2024)
- [MolScribe: Robust Molecular Structure Recognition with Image-To-Graph Generation](https://mlsanity.com/paper/2205.14311/) (2022)
- [Rethinking Tokenizer and Decoder in Masked Graph Modeling for Molecules](https://mlsanity.com/paper/2310.14753/) (2023)
- [A Group Symmetric Stochastic Differential Equation Model for Molecule Multi-modal Pretraining](https://mlsanity.com/paper/2305.18407/) (2023)
- [Molecular Structure Extraction From Documents Using Deep Learning](https://mlsanity.com/paper/1802.04903/) (2018)
- [MolGrapher: Graph-based Visual Recognition of Chemical Structures](https://mlsanity.com/paper/2308.12234/) (2023)
- [Multimodal Molecular Pretraining via Modality Blending](https://mlsanity.com/paper/2307.06235/) (2023)
- [ChemGrapher: Optical Graph Recognition of Chemical Compounds by Deep Learning](https://mlsanity.com/paper/2002.09914/) (2020)
- [UniCorn: A Unified Contrastive Learning Approach for Multi-view Molecular Representation Learning](https://mlsanity.com/paper/2405.10343/) (2024)
- [MolNexTR: A Generalized Deep Learning Model for Molecular Image Recognition](https://mlsanity.com/paper/2403.03691/) (2024)
- [MolMiner: You only look once for chemical structure recognition](https://mlsanity.com/paper/2205.11016/) (2022)
- [IMG2SMI: Translating Molecular Structure Images to Simplified Molecular-input Line-entry System](https://mlsanity.com/paper/2109.04202/) (2021)
- [Image-to-Graph Transformers for Chemical Structure Recognition](https://mlsanity.com/paper/2202.09580/) (2022)

## Built on by (most cited first, 17 of 17)

- [GTR-CoT: Graph Traversal as Visual Chain of Thought for Molecular Structure Recognition](https://mlsanity.com/paper/2506.07553/) (2025)
- [RxnBench: A Multimodal Benchmark for Evaluating Large Language Models on Chemical Reaction Understanding from Scientific Literature](https://mlsanity.com/paper/2512.23565/) (2025)
- [Uni-Parser Technical Report](https://mlsanity.com/paper/2512.15098/) (2025)
- [Doc2SAR: A Synergistic Framework for High-Fidelity Extraction of Structure-Activity Relationships from Scientific Documents](https://mlsanity.com/paper/2506.21625/) (2025)
- [MolSight: Optical Chemical Structure Recognition with SMILES Pretraining, Multi-Granularity Learning and Reinforcement Learning](https://mlsanity.com/paper/2511.17300/) (2025)
- [MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures](https://mlsanity.com/paper/2603.28550/) (2026)
- [From Human Labels to Literature: Semi-Supervised Learning of NMR Chemical Shifts at Scale](https://mlsanity.com/paper/2601.18524/) (2026)
- [BioMiner: A Multi-modal System for Automated Mining of Protein-Ligand Bioactivity Data from Literature](https://mlsanity.com/paper/2604.21508/) (2026)
- [MolRecBench-Wild: A Real-World Benchmark for Optical Chemical Structure Recognition](https://mlsanity.com/paper/2605.05832/) (2026)
- [MoleCode unlocks structural intelligence in large language models](https://mlsanity.com/paper/2605.16480/) (2026)
- [MarkushGlyph and OCSRGlyph: Improved Chemical Structure Recognition](https://mlsanity.com/paper/2607.28532/) (2026)
- [MinerU.Chem: A High-Precision System for Optical Chemical Structure and Reaction Recognition](https://mlsanity.com/paper/2608.03525/) (2026)
- [SubGrapher: Visual Fingerprinting of Chemical Structures](https://mlsanity.com/paper/2504.19695/) (2025)
- [COMO: Closed-Loop Optical Molecule Recognition with Minimum Risk Training](https://mlsanity.com/paper/2604.23546/) (2026)
- [MolParser-Mobile: Ultrafast OCSR System for Large-Scale Chemical Literature Mining](https://mlsanity.com/paper/2609.05807/) (2026)
- [ViCoR: Reliable Molecular Structure Extraction via Spatially Aligned Verification and Executable Revision](https://mlsanity.com/paper/2609.33603/) (2026)
- [R-GroundBench: A Diagnostic Benchmark for R-Group Groundingin Markush Molecular Editing](https://mlsanity.com/paper/2610.00700/) (2026)

## Abstract (arXiv)

In recent decades, chemistry publications and patents have increased rapidly. A significant portion of key information is embedded in molecular structure figures, complicating large-scale literature searches and limiting the application of large language models in fields such as biology, chemistry, and pharmaceuticals. The automatic extraction of precise chemical structures is of critical importance. However, the presence of numerous Markush structures in real-world documents, along with variations in molecular image quality, drawing styles, and noise, significantly limits the performance of existing optical chemical structure recognition (OCSR) methods. We present MolParser, a novel end-to-end OCSR method that efficiently and accurately recognizes chemical structures from real-world documents, including difficult Markush structure. We use a extended SMILES encoding rule to annotate our training dataset. Under this rule, we build MolParser-7M, the largest annotated molecular image dataset to our knowledge. While utilizing a large amount of synthetic data, we employed active learning methods to incorporate substantial in-the-wild data, specifically samples cropped from real patents and scientific literature, into the training process. We trained an end-to-end molecular image captioning model, MolParser, using a curriculum learning approach. MolParser significantly outperforms classical and learning-based methods across most scenarios, with potential for broader downstream applications. The dataset is publicly available in huggingface.