# SmileyLlama: Modifying Large Language Models for Directed Chemical Space Exploration

arXiv: https://arxiv.org/abs/2409.02231 · HTML: https://mlsanity.com/paper/2409.02231/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Preference optimization for molecular design, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 68 of its 140 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Preference optimization for molecular design](https://mlsanity.com/alignment/preference-optimization-for-molecular-design/)

## Builds on (most cited first, 30 of 68)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale](https://mlsanity.com/paper/2207.00032/) (2022)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [MolGAN: An implicit generative model for small molecular graphs](https://mlsanity.com/paper/1805.11973/) (2018)
- [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://mlsanity.com/paper/2101.06840/) (2021)
- [GuacaMol: Benchmarking Models for De Novo Molecular Design](https://mlsanity.com/paper/1811.09621/) (2018)
- [Molecular De Novo Design through Deep Reinforcement Learning](https://mlsanity.com/paper/1704.07555/) (2017)

## Built on by (most cited first, 14 of 14)

- [Hypothesis Generation for Materials Discovery and Design Using Goal-Driven and Constraint-Guided LLM Agents](https://mlsanity.com/paper/2501.13299/) (2025)
- [Speak-to-Structure: Evaluating LLMs in Open-domain Natural Language-Driven Molecule Generation](https://mlsanity.com/paper/2412.14642/) (2024)
- [MatterChat: A Multi-Modal LLM for Material Science](https://mlsanity.com/paper/2502.13107/) (2025)
- [A Survey of Large Language Models for Text-Guided Molecular Discovery: from Molecule Generation to Optimization](https://mlsanity.com/paper/2505.16094/) (2025)
- [SynLlama: Generating Synthesizable Molecules and Their Analogs with Large Language Models](https://mlsanity.com/paper/2503.12602/) (2025)
- [Atom-anchored LLMs speak Chemistry: A Retrosynthesis Demonstration](https://mlsanity.com/paper/2510.16590/) (2025)
- [Mol-MoE: Training Preference-Guided Routers for Molecule Generation](https://mlsanity.com/paper/2502.05633/) (2025)
- [M^4olGen: Multi-Agent, Multi-Stage Molecular Generation under Precise Multi-Property Constraints](https://mlsanity.com/paper/2601.10131/) (2026)
- [Towards Agentic Intelligence for Materials Science](https://mlsanity.com/paper/2602.00169/) (2026)
- [Towards Diverse Scientific Hypothesis Search with Large Language Models](https://mlsanity.com/paper/2606.10587/) (2026)
- [COGTRL: Training LLMs for Scientific Discovery Assistance using Cognitive Traces via Reinforcement Learning](https://mlsanity.com/paper/2608.30109/) (2026)
- [DrugPlayGround: Benchmarking Large Language Models and Embeddings for Drug Discovery](https://mlsanity.com/paper/2604.02346/) (2026)
- [EvoMD-LLM: Learning the Language of Species Evolution in Reactive Molecular Dynamics](https://mlsanity.com/paper/2605.29394/) (2026)
- [ElemeNet: Multiscale Molecular Machine Learning with Uncertainty Quantification Across the Periodic Table](https://mlsanity.com/paper/2606.30961/) (2026)

## Abstract (arXiv)

We show that large language model (LLMs) can be transformed via supervised fine-tuning (SFT) of engineered prompts into SmileyLlama for exploring the chemical space of drug molecules. We benchmark SmileyLlama against pre-trained LLMs and chemical language models (CLM) trained from scratch for generating valid and novel drug-like molecules, and use direct preference optimization (DPO) to both improve SmileyLlama's adherence to a prompt and as part of the iMiner reinforcement learning framework to predict molecules with optimized 3D conformations and high binding affinity to drug targets. By training an LLM to speak directly as a CLM, while retaining most of its natural language capabilities, we show that we can reliably generate molecules with user-specified properties rather than acting only as a chatbot with knowledge of chemistry or as a virtual assistant. While SmileyLlama is geared toward drug discovery, the SFT/DPO/LLM framework can be extended to other chemical, biological, and materials applications.