# Uni-Mol2: Exploring Molecular Pretraining Model at Scale

arXiv: https://arxiv.org/abs/2406.14969 · HTML: https://mlsanity.com/paper/2406.14969/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 53 references that are in the map.
- First posted on arXiv 21 Jun 2024.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [SchNet: A continuous-filter convolutional neural network for modeling quantum interactions](https://mlsanity.com/paper/1706.08566/) (2017)
- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)
- [Pre-training Molecular Graph Representation with 3D Geometry](https://mlsanity.com/paper/2110.07728/) (2021)
- [ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Prediction](https://mlsanity.com/paper/2106.06130/) (2021)
- [Spherical Message Passing for 3D Graph Networks](https://mlsanity.com/paper/2102.05013/) (2021)
- [Motif-based Graph Self-Supervised Learning for Molecular Property Prediction](https://mlsanity.com/paper/2110.00987/) (2021)
- [UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining](https://mlsanity.com/paper/2304.09151/) (2023)
- [3D Infomax improves GNNs for Molecular Property Prediction](https://mlsanity.com/paper/2110.04126/) (2021)
- [Benchmarking Graphormer on Large-Scale Molecular Modeling Datasets](https://mlsanity.com/paper/2203.04810/) (2022)
- [Graph-based Molecular Representation Learning](https://mlsanity.com/paper/2207.04869/) (2022)
- [Unraveling the Mystery of Scaling Laws: Part I](https://mlsanity.com/paper/2403.06563/) (2024)
- [Uncovering Neural Scaling Laws in Molecular Representation Learning](https://mlsanity.com/paper/2309.15123/) (2023)
- [Highly Accurate Quantum Chemical Property Prediction with Uni-Mol+](https://mlsanity.com/paper/2303.16982/) (2023)

## Built on by (most cited first, 20 of 20)

- [Enhancing the Scalability and Applicability of Kohn-Sham Hamiltonians for Molecular Systems](https://mlsanity.com/paper/2502.19227/) (2025)
- [Graph-structured Small Molecule Drug Discovery Through Deep Learning: Progress, Challenges, and Opportunities](https://mlsanity.com/paper/2502.08975/) (2025)
- [GeoRecon: Graph-Level Representation Learning for 3D Molecules via Reconstruction-Based Pretraining](https://mlsanity.com/paper/2506.13174/) (2025)
- [SSTAG: Structure-Aware Self-Supervised Learning Method for Text-Attributed Graphs](https://mlsanity.com/paper/2510.01248/) (2025)
- [Scaling Structure Aware Virtual Screening to Billions of Molecules with SPRINT](https://mlsanity.com/paper/2411.15418/) (2024)
- [Uni-Mol3: A Multi-Molecular Foundation Model for Advancing Organic Reaction Modeling](https://mlsanity.com/paper/2508.00920/) (2025)
- [Structure-Aware Fusion with Progressive Injection for Multimodal Molecular Representation Learning](https://mlsanity.com/paper/2510.23640/) (2025)
- [ChemAmp: Amplified Chemistry Tools via Composable Agents](https://mlsanity.com/paper/2505.21569/) (2025)
- [A Comprehensive Review of Transformer-based language models for Protein Sequence Analysis and Design](https://mlsanity.com/paper/2507.13646/) (2025)
- [MolSculpt: Sculpting 3D Molecular Geometries from Chemical Syntax](https://mlsanity.com/paper/2512.10991/) (2025)
- [Entropy-Guided Dynamic Tokens for Graph-LLM Alignment in Molecular Understanding](https://mlsanity.com/paper/2602.02742/) (2026)
- [Suiren-1.0 Technical Report: A Family of Molecular Foundation Models](https://mlsanity.com/paper/2603.21942/) (2026)
- [Multimodal 3D Genome Pre-training](https://mlsanity.com/paper/2504.09060/) (2025)
- [MolGA: Molecular Graph Adaptation with Pre-trained 2D Graph Encoder](https://mlsanity.com/paper/2510.07289/) (2025)
- [3D-GSRD: 3D Molecular Graph Auto-Encoder with Selective Re-mask Decoding](https://mlsanity.com/paper/2510.16780/) (2025)
- [MultiPUFFIN: A Multimodal Domain-Constrained Foundation Model for Molecular Property Prediction of Small Molecules](https://mlsanity.com/paper/2603.00857/) (2026)
- [DrugPlayGround: Benchmarking Large Language Models and Embeddings for Drug Discovery](https://mlsanity.com/paper/2604.02346/) (2026)
- [SC3: The Multi-Solvent Solubility Challenge and Benchmark](https://mlsanity.com/paper/2606.07656/) (2026)
- [Retrieval-Augmented Multimodal Learning for Enzyme-Substrate Interaction Prediction Under Low-Homology Shift](https://mlsanity.com/paper/2606.22823/) (2026)
- [S1-Omni: A Unified Multimodal Reasoning Model for Scientific Understanding, Prediction, and Generation](https://mlsanity.com/paper/2607.15686/) (2026)

## Abstract (arXiv)

In recent years, pretraining models have made significant advancements in the fields of natural language processing (NLP), computer vision (CV), and life sciences. The significant advancements in NLP and CV are predominantly driven by the expansion of model parameters and data size, a phenomenon now recognized as the scaling laws. However, research exploring scaling law in molecular pretraining models remains unexplored. In this work, we present Uni-Mol2 , an innovative molecular pretraining model that leverages a two-track transformer to effectively integrate features at the atomic level, graph level, and geometry structure level. Along with this, we systematically investigate the scaling law within molecular pretraining models, characterizing the power-law correlations between validation loss and model size, dataset size, and computational resources. Consequently, we successfully scale Uni-Mol2 to 1.1 billion parameters through pretraining on 800 million conformations, making it the largest molecular pretraining model to date. Extensive experiments show consistent improvement in the downstream tasks as the model size grows. The Uni-Mol2 with 1.1B parameters also outperforms existing methods, achieving an average 27% improvement on the QM9 and 14% on COMPAS-1D dataset.