# textttMiniMol: A Parameter-Efficient Foundation Model for Molecular Learning

arXiv: https://arxiv.org/abs/2404.14986 · HTML: https://mlsanity.com/paper/2404.14986/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 50 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Builds on (most cited first, 27 of 27)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Recipe for a General, Powerful, Scalable Graph Transformer](https://mlsanity.com/paper/2205.12454/) (2022)
- [Directional Message Passing for Molecular Graphs](https://mlsanity.com/paper/2003.03123/) (2020)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs](https://mlsanity.com/paper/2103.09430/) (2021)
- [Do Transformers Really Perform Bad for Graph Representation?](https://mlsanity.com/paper/2106.05234/) (2021)
- [Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development](https://mlsanity.com/paper/2102.09548/) (2021)
- [Fine-tuning Language Models for Factuality](https://mlsanity.com/paper/2311.08401/) (2023)
- [Large-Scale Chemical Language Representations Capture Molecular Structure and Properties](https://mlsanity.com/paper/2106.09553/) (2021)
- [Graph neural networks for materials science and chemistry](https://mlsanity.com/paper/2208.09481/) (2022)
- [ChemBERTa-2: Towards Chemical Foundation Models](https://mlsanity.com/paper/2209.01712/) (2022)
- [Graph Foundation Models: Concepts, Opportunities and Challenges](https://mlsanity.com/paper/2310.11829/) (2023)
- [SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery](https://mlsanity.com/paper/1911.04738/) (2019)
- [MolFM: A Multimodal Molecular Foundation Model](https://mlsanity.com/paper/2307.09484/) (2023)
- [An Introduction to Electrocatalyst Design using Machine Learning for Renewable Energy Storage](https://mlsanity.com/paper/2010.09435/) (2020)
- [Learning Graph-Level Representation for Drug Discovery](https://mlsanity.com/paper/1709.03741/) (2017)
- [Distributed Inference and Fine-tuning of Large Language Models Over The Internet](https://mlsanity.com/paper/2312.08361/) (2023)
- [Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasets](https://mlsanity.com/paper/2310.04292/) (2023)
- [From Molecules to Materials: Pre-training Large Generalizable Models for Atomic Property Prediction](https://mlsanity.com/paper/2310.16802/) (2023)
- [AtomNet: A Deep Convolutional Neural Network for Bioactivity Prediction in Structure-based Drug Discovery](https://mlsanity.com/paper/1510.02855/) (2015)
- [Highly Accurate Quantum Chemical Property Prediction with Uni-Mol+](https://mlsanity.com/paper/2303.16982/) (2023)
- [MolE: a molecular foundation model for drug discovery](https://mlsanity.com/paper/2211.02657/) (2022)

## Built on by (most cited first, 20 of 20)

- [Position: Graph Learning Will Lose Relevance Due To Poor Benchmarks](https://mlsanity.com/paper/2502.14546/) (2025)
- [Fully-inductive Node Classification on Arbitrary Graphs](https://mlsanity.com/paper/2405.20445/) (2024)
- [Accelerating Scientific Discovery with Autonomous Goal-evolving Agents](https://mlsanity.com/paper/2512.21782/) (2025)
- [Beyond Efficiency: Molecular Data Pruning for Enhanced Generalization](https://mlsanity.com/paper/2409.01081/) (2024)
- [Towards Graph Foundation Models: A Transferability Perspective](https://mlsanity.com/paper/2503.09363/) (2025)
- [Tabular foundation models for in-context prediction of molecular properties](https://mlsanity.com/paper/2604.16123/) (2026)
- [Molecular Machine Learning in Chemical Process Design](https://mlsanity.com/paper/2508.20527/) (2025)
- [Pretraining Graph Transformers with Atom-in-a-Molecule Quantum Properties for Improved ADMET Modeling](https://mlsanity.com/paper/2410.08024/) (2024)
- [MiNT: Multi-Network Training for Transfer Learning on Temporal Graphs](https://mlsanity.com/paper/2406.10426/) (2024)
- [Breaking Bad Molecules: Are MLLMs Ready for Structure-Level Molecular Detoxification?](https://mlsanity.com/paper/2506.10912/) (2025)
- [Can Molecular Foundation Models Know What They Don't Know? A Simple Remedy with Preference Optimization](https://mlsanity.com/paper/2509.25509/) (2025)
- [CAGenMol: Condition-Aware Diffusion Language Model for Goal-Directed Molecular Generation](https://mlsanity.com/paper/2604.11483/) (2026)
- [Riemannian Geometry Speaks Louder Than Words: From Graph Foundation Model to Next-Generation Graph Intelligence](https://mlsanity.com/paper/2603.21601/) (2026)
- [Monroe: A Molecular Foundation Model for In-Context Probabilistic Inference](https://mlsanity.com/paper/2608.18982/) (2026)
- [Amortized Molecular Optimization via Group Relative Policy Optimization](https://mlsanity.com/paper/2602.12162/) (2026)
- [A Systematic Evaluation of Co-folding Model Representations for Small-Molecule Learning](https://mlsanity.com/paper/2602.13249/) (2026)
- [Self-Driving Datasets: From 20 Million Papers to Nuanced Biomedical Knowledge at Scale](https://mlsanity.com/paper/2605.07022/) (2026)
- [GLACIER: A Multimodal Student-Teacher Foundation Model for Molecular Property Prediction](https://mlsanity.com/paper/2606.11382/) (2026)
- [Molecular Property Prediction under Structural Shift with Tabular Foundation Models](https://mlsanity.com/paper/2609.38744/) (2026)
- [Does Pretraining Teach Chemical Language Models Chemistry?](https://mlsanity.com/paper/2610.18651/) (2026)

## Abstract (arXiv)

In biological tasks, data is rarely plentiful as it is generated from hard-to-gather measurements. Therefore, pre-training foundation models on large quantities of available data and then transfer to low-data downstream tasks is a promising direction. However, how to design effective foundation models for molecular learning remains an open question, with existing approaches typically focusing on models with large parameter capacities. In this work, we propose textttMiniMol, a foundational model for molecular learning with 10 million parameters. textttMiniMol is pre-trained on a mix of roughly 3300 sparsely defined graph- and node-level tasks of both quantum and biological nature. The pre-training dataset includes approximately 6 million molecules and 500 million labels. To demonstrate the generalizability of textttMiniMol across tasks, we evaluate it on downstream tasks from the Therapeutic Data Commons (TDC) ADMET group showing significant improvements over the prior state-of-the-art foundation model across 17 tasks. textttMiniMol will be a public and open-sourced model for future research.