# LICO: Large Language Models for In-Context Molecular Optimization

arXiv: https://arxiv.org/abs/2406.18851 · HTML: https://mlsanity.com/paper/2406.18851/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 74 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 30 of 51)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [GuacaMol: Benchmarking Models for De Novo Molecular Design](https://mlsanity.com/paper/1811.09621/) (2018)
- [GFlowNet Foundations](https://mlsanity.com/paper/2111.09266/) (2021)
- [Molecular De Novo Design through Deep Reinforcement Learning](https://mlsanity.com/paper/1704.07555/) (2017)
- [Pre-Trained Language Models for Interactive Decision-Making](https://mlsanity.com/paper/2202.01771/) (2022)
- [Large Language Models as General Pattern Machines](https://mlsanity.com/paper/2307.04721/) (2023)
- [Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization](https://mlsanity.com/paper/2206.12411/) (2022)
- [LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks](https://mlsanity.com/paper/2206.06565/) (2022)
- [Large Language Models to Enhance Bayesian Optimization](https://mlsanity.com/paper/2402.03921/) (2024)
- [Evolution through Large Models](https://mlsanity.com/paper/2206.08896/) (2022)
- [EvoPrompting: Language Models for Code-Level Neural Architecture Search](https://mlsanity.com/paper/2302.14838/) (2023)
- [Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling](https://mlsanity.com/paper/2207.04179/) (2022)

## Built on by (most cited first, 26 of 26)

- [Sequential Large Language Model-Based Hyper-parameter Optimization](https://mlsanity.com/paper/2410.20302/) (2024)
- [Language-Based Bayesian Optimization Research Assistant (BORA)](https://mlsanity.com/paper/2501.16224/) (2025)
- [LLMs for Bayesian Optimization in Scientific Domains: Are We There Yet?](https://mlsanity.com/paper/2509.21403/) (2025)
- [A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving](https://mlsanity.com/paper/2509.08269/) (2025)
- [Language Model Embeddings Can Be Sufficient for Bayesian Optimization](https://mlsanity.com/paper/2410.10190/) (2024)
- [ExLLM: Experience-Enhanced LLM Optimization for Molecular Design and Beyond](https://mlsanity.com/paper/2502.12845/) (2025)
- [Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings](https://mlsanity.com/paper/2506.07109/) (2025)
- [Small Molecule Optimization with Large Language Models](https://mlsanity.com/paper/2407.18897/) (2024)
- [Reference-guided Policy Optimization for Molecular Optimization via LLM Reasoning](https://mlsanity.com/paper/2603.05900/) (2026)
- [Tree-Structured Synergy of Large Language Models and Bayesian Optimization for Efficient CASH](https://mlsanity.com/paper/2601.12355/) (2026)
- [Diffusion Large Language Models for Black-Box Optimization](https://mlsanity.com/paper/2601.14446/) (2026)
- [Iceberg: Enhancing HLS Modeling with Synthetic Data](https://mlsanity.com/paper/2507.09948/) (2025)
- [Coder as Editor: Code-driven Interpretable Molecular Optimization](https://mlsanity.com/paper/2510.14455/) (2025)
- [SEISMO: Explanation-Aware, Trajectory-Conditioned LLM Agents for Sample-Efficient Molecular Optimisation](https://mlsanity.com/paper/2602.00663/) (2026)
- [Agents on a Tree: Pathwise Coordination for Multi-Objective Molecular Optimization](https://mlsanity.com/paper/2606.00008/) (2026)
- [Sample Efficient Generative Optimization for Molecular Design](https://mlsanity.com/paper/2607.12488/) (2026)
- [Assay2Mol: large language model-based drug design using BioAssay context](https://mlsanity.com/paper/2507.12574/) (2025)
- [INDIBATOR: Diverse and Fact-Grounded Individuality for Multi-Agent Debate in Molecular Discovery](https://mlsanity.com/paper/2602.01815/) (2026)
- [MolLIBRA: Genetic Molecular Optimization with Multi-Fingerprint Surrogates and Text-Molecule Aligned Critic](https://mlsanity.com/paper/2602.07002/) (2026)
- [Scaffold-Conditioned Preference Triplets for Controllable Molecular Optimization with Large Language Models](https://mlsanity.com/paper/2604.12350/) (2026)
- [C-MORAL: Controllable Multi-Objective Molecular Optimization with Reinforcement Alignment for LLMs](https://mlsanity.com/paper/2604.23061/) (2026)
- [FORGE: Fragment-Oriented Ranking and Generation for Context-Aware Molecular Optimization](https://mlsanity.com/paper/2605.10230/) (2026)
- [MolDeTox: Evaluating Language Model's Stepwise Fragment Editing for Molecular Detoxification](https://mlsanity.com/paper/2605.12181/) (2026)
- [LipoAgent: Coordinating Fine-Tuned LLM Agents for Safer Lipid Design](https://mlsanity.com/paper/2605.25250/) (2026)
- [Agentic Molecular Recovery via Molecule-Aware Exploration](https://mlsanity.com/paper/2606.05847/) (2026)
- [Frontier LLMs are effective batch optimizers: Assessing reasoning models in continuous and discrete settings](https://mlsanity.com/paper/2609.03177/) (2026)

## Abstract (arXiv)

Optimizing black-box functions is a fundamental problem in science and engineering. To solve this problem, many approaches learn a surrogate function that estimates the underlying objective from limited historical evaluations. Large Language Models (LLMs), with their strong pattern-matching capabilities via pretraining on vast amounts of data, stand out as a potential candidate for surrogate modeling. However, directly prompting a pretrained language model to produce predictions is not feasible in many scientific domains due to the scarcity of domain-specific data in the pretraining corpora and the challenges of articulating complex problems in natural language. In this work, we introduce LICO, a general-purpose model that extends arbitrary base LLMs for black-box optimization, with a particular application to the molecular domain. To achieve this, we equip the language model with a separate embedding layer and prediction layer, and train the model to perform in-context predictions on a diverse set of functions defined over the domain. Once trained, LICO can generalize to unseen molecule properties simply via in-context prompting. LICO performs competitively on PMO, a challenging molecular optimization benchmark comprising 23 objective functions, and achieves state-of-the-art performance on its low-budget version PMO-1K.