# MolNexTR: A Generalized Deep Learning Model for Molecular Image Recognition

arXiv: https://arxiv.org/abs/2403.03691 · HTML: https://mlsanity.com/paper/2403.03691/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 43 references that are in the map.
- First posted on arXiv 6 Mar 2024.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [Relationformer: A Unified Framework for Image-to-Graph Generation](https://mlsanity.com/paper/2203.10202/) (2022)
- [MolScribe: Robust Molecular Structure Recognition with Image-To-Graph Generation](https://mlsanity.com/paper/2205.14311/) (2022)
- [Molecular Structure Extraction From Documents Using Deep Learning](https://mlsanity.com/paper/1802.04903/) (2018)
- [ChemGrapher: Optical Graph Recognition of Chemical Compounds by Deep Learning](https://mlsanity.com/paper/2002.09914/) (2020)
- [MolMiner: You only look once for chemical structure recognition](https://mlsanity.com/paper/2205.11016/) (2022)
- [Image-to-Graph Transformers for Chemical Structure Recognition](https://mlsanity.com/paper/2202.09580/) (2022)

## Built on by (most cited first, 16 of 16)

- [MolParser: End-to-end Visual Recognition of Molecule Structures in the Wild](https://mlsanity.com/paper/2411.11098/) (2024)
- [ChemDFM-X: Towards Large Multimodal Model for Chemistry](https://mlsanity.com/paper/2409.13194/) (2024)
- [From Generalist to Specialist: A Survey of Large Language Models for Chemistry](https://mlsanity.com/paper/2412.19994/) (2024)
- [MarkushGrapher: Joint Visual and Textual Recognition of Markush Structures](https://mlsanity.com/paper/2503.16096/) (2025)
- [Uni-Parser Technical Report](https://mlsanity.com/paper/2512.15098/) (2025)
- [MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures](https://mlsanity.com/paper/2603.28550/) (2026)
- [Large Language Models Transform Organic Synthesis From Reaction Prediction to Automation](https://mlsanity.com/paper/2508.05427/) (2025)
- [BioMiner: A Multi-modal System for Automated Mining of Protein-Ligand Bioactivity Data from Literature](https://mlsanity.com/paper/2604.21508/) (2026)
- [MolRecBench-Wild: A Real-World Benchmark for Optical Chemical Structure Recognition](https://mlsanity.com/paper/2605.05832/) (2026)
- [MarkushGlyph and OCSRGlyph: Improved Chemical Structure Recognition](https://mlsanity.com/paper/2607.28532/) (2026)
- [MinerU.Chem: A High-Precision System for Optical Chemical Structure and Reaction Recognition](https://mlsanity.com/paper/2608.03525/) (2026)
- [Real Data Closes Synthetic-to-Real Gap in Optical Chemical Structure Recognition](https://mlsanity.com/paper/2608.09100/) (2026)
- [VERDICT: Agreement Beats Pixel-Space Verification in Real-Document OCSR](https://mlsanity.com/paper/2608.22183/) (2026)
- [Differential Learning for Robust Prediction of Thermal Stability with Application to Energetic Materials](https://mlsanity.com/paper/2608.23874/) (2026)
- [MolParser-Mobile: Ultrafast OCSR System for Large-Scale Chemical Literature Mining](https://mlsanity.com/paper/2609.05807/) (2026)
- [ViCoR: Reliable Molecular Structure Extraction via Spatially Aligned Verification and Executable Revision](https://mlsanity.com/paper/2609.33603/) (2026)

## Abstract (arXiv)

In the field of chemical structure recognition, the task of converting molecular images into machine-readable data formats such as SMILES string stands as a significant challenge, primarily due to the varied drawing styles and conventions prevalent in chemical literature. To bridge this gap, we proposed MolNexTR, a novel image-to-graph deep learning model that collaborates to fuse the strengths of ConvNext, a powerful Convolutional Neural Network variant, and Vision-TRansformer. This integration facilitates a more detailed extraction of both local and global features from molecular images. MolNexTR can predict atoms and bonds simultaneously and understand their layout rules. It also excels at flexibly integrating symbolic chemistry principles to discern chirality and decipher abbreviated structures. We further incorporate a series of advanced algorithms, including an improved data augmentation module, an image contamination module, and a post-processing module for getting the final SMILES output. These modules cooperate to enhance the model's robustness to diverse styles of molecular images found in real literature. In our test sets, MolNexTR has demonstrated superior performance, achieving an accuracy rate of 81-97%, marking a significant advancement in the domain of molecular structure recognition.