# nach0: Multimodal Natural and Chemical Languages Foundation Model

arXiv: https://arxiv.org/abs/2311.12410 · HTML: https://mlsanity.com/paper/2311.12410/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 85 references that are in the map.
- First posted on arXiv 21 Nov 2023.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models](https://mlsanity.com/paper/1811.12823/) (2018)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction](https://mlsanity.com/paper/1811.02633/) (2018)
- [LinkBERT: Pretraining Language Models with Document Links](https://mlsanity.com/paper/2203.15827/) (2022)
- [Translation between Molecules and Natural Language](https://mlsanity.com/paper/2204.11817/) (2022)
- [Does Synthetic Data Generation of LLMs Help Clinical Text Mining?](https://mlsanity.com/paper/2303.04360/) (2023)
- [Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models](https://mlsanity.com/paper/2306.08018/) (2023)
- [SciFive: a text-to-text transformer model for biomedical literature](https://mlsanity.com/paper/2106.03598/) (2021)
- [Molecular representation learning with language models and domain-relevant auxiliary tasks](https://mlsanity.com/paper/2011.13230/) (2020)
- [BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model](https://mlsanity.com/paper/2204.03905/) (2022)
- [A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature](https://mlsanity.com/paper/1806.04185/) (2018)
- [BioMegatron: Larger Biomedical Domain Language Model](https://mlsanity.com/paper/2010.06060/) (2020)
- [SELFIES and the future of molecular string representations](https://mlsanity.com/paper/2204.00056/) (2022)

## Built on by (most cited first, 26 of 26)

- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning](https://mlsanity.com/paper/2502.02871/) (2025)
- [LICO: Large Language Models for In-Context Molecular Optimization](https://mlsanity.com/paper/2406.18851/) (2024)
- [MolTC: Towards Molecular Relational Modeling In Language Models](https://mlsanity.com/paper/2402.03781/) (2024)
- [Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Survey](https://mlsanity.com/paper/2403.01528/) (2024)
- [From Words to Molecules: A Survey of Large Language Models in Chemistry](https://mlsanity.com/paper/2402.01439/) (2024)
- [PRESTO: Progressive Pretraining Enhances Synthetic Chemistry Outcomes](https://mlsanity.com/paper/2406.13193/) (2024)
- [TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewards](https://mlsanity.com/paper/2512.07761/) (2025)
- [From Generalist to Specialist: A Survey of Large Language Models for Chemistry](https://mlsanity.com/paper/2412.19994/) (2024)
- [mCLM: A Modular Chemical Language Model that Generates Functional and Makeable Molecules](https://mlsanity.com/paper/2505.12565/) (2025)
- [A Survey of AI for Materials Science: Foundation Models, LLM Agents, Datasets, and Tools](https://mlsanity.com/paper/2506.20743/) (2025)
- [Scalable and Cost-Efficient de Novo Template-Based Molecular Generation](https://mlsanity.com/paper/2506.19865/) (2025)
- [MolVision: Molecular Property Prediction with Vision Language Models](https://mlsanity.com/paper/2507.03283/) (2025)
- [Omni-Mol: Multitask Molecular Model for Any-to-any Modalities](https://mlsanity.com/paper/2502.01074/) (2025)
- [Retro-Expert: Collaborative Reasoning for Interpretable Retrosynthesis](https://mlsanity.com/paper/2508.10967/) (2025)
- [nach0-pc: Multi-task Language Model with Molecular Point Cloud Encoder](https://mlsanity.com/paper/2410.09240/) (2024)
- [Towards Foundation Model for Chemical Reactor Modeling: Meta-Learning with Physics-Informed Adaptation](https://mlsanity.com/paper/2405.11752/) (2024)
- [Foundation Models for AI-Enabled Biological Design](https://mlsanity.com/paper/2505.11610/) (2025)
- [Large Language Models Transform Organic Synthesis From Reaction Prediction to Automation](https://mlsanity.com/paper/2508.05427/) (2025)
- [Machine Learning Predicts Upper Secondary Education Dropout as Early as the End of Primary School](https://mlsanity.com/paper/2403.14663/) (2024)
- [Modular Multi-Task Learning for Chemical Reaction Prediction](https://mlsanity.com/paper/2602.10404/) (2026)
- [MMAI Gym for Science: Training Liquid Foundation Models for Drug Discovery](https://mlsanity.com/paper/2603.03517/) (2026)
- [Do Language Models Dream of Binding Molecules? Benchmarking LLMs under Spatial Constraints](https://mlsanity.com/paper/2607.18144/) (2026)
- [Multi-Granular Rationale-Guided Molecular LLM for Property Prediction](https://mlsanity.com/paper/2608.10480/) (2026)
- [Training Chemical Plausibility-Aware Large Language Models for Single-Step Retrosynthesis](https://mlsanity.com/paper/2608.18940/) (2026)
- [MolSC: Leveraging Substituent Contributions to Enhance Fine-grained Molecular Understanding in LLMs](https://mlsanity.com/paper/2609.23073/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have substantially driven scientific progress in various domains, and many papers have demonstrated their ability to tackle complex problems with creative solutions. Our paper introduces a new foundation model, nach0, capable of solving various chemical and biological tasks: biomedical question answering, named entity recognition, molecular generation, molecular synthesis, attributes prediction, and others. nach0 is a multi-domain and multi-task encoder-decoder LLM pre-trained on unlabeled text from scientific literature, patents, and molecule strings to incorporate a range of chemical and linguistic knowledge. We employed instruction tuning, where specific task-related instructions are utilized to fine-tune nach0 for the final set of tasks. To train nach0 effectively, we leverage the NeMo framework, enabling efficient parallel optimization of both base and large model versions. Extensive experiments demonstrate that our model outperforms state-of-the-art baselines on single-domain and cross-domain tasks. Furthermore, it can generate high-quality outputs in molecular and textual formats, showcasing its effectiveness in multi-domain setups.