# Atom-by-atom protein generation and beyond with language models

arXiv: https://arxiv.org/abs/2308.09482 · HTML: https://mlsanity.com/paper/2308.09482/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 28 references that are in the map.
- First posted on arXiv 16 Aug 2023.

## Builds on (most cited first, 8 of 8)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Keeping it Simple: Language Models can learn Complex Molecular Distributions](https://mlsanity.com/paper/2112.03041/) (2021)
- [Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB files](https://mlsanity.com/paper/2305.05708/) (2023)
- [Self-Referencing Embedded Strings (SELFIES): A 100% robust molecular string representation](https://mlsanity.com/paper/1905.13741/) (2019)
- [Scalable Fragment-Based 3D Molecular Design with Reinforcement Learning](https://mlsanity.com/paper/2202.00658/) (2022)
- [Bayesian Variational Optimization for Combinatorial Spaces](https://mlsanity.com/paper/2011.02004/) (2020)

## Built on by (most cited first, 2 of 2)

- [Fine-Tuned Language Models Generate Stable Inorganic Materials as Text](https://mlsanity.com/paper/2402.04379/) (2024)
- [RLDBF: Enhancing LLMs Via Reinforcement Learning With DataBase FeedBack](https://mlsanity.com/paper/2504.03713/) (2025)

## Abstract (arXiv)

Protein language models learn powerful representations directly from sequences of amino acids. However, they are constrained to generate proteins with only the set of amino acids represented in their vocabulary. In contrast, chemical language models learn atom-level representations of smaller molecules that include every atom, bond, and ring. In this work, we show that chemical language models can learn atom-level representations of proteins enabling protein generation unconstrained to the standard genetic code and far beyond it. In doing so, we show that language models can generate entire proteins atom by atom -- effectively learning the multiple hierarchical layers of molecular information that define proteins from their primary sequence to their secondary, and tertiary structure. We demonstrate language models are able to explore beyond protein space -- generating proteins with modified sidechains that form unnatural amino acids. Even further, we find that language models can explore chemical space and protein space simultaneously and generate novel examples of protein-drug conjugates. The results demonstrate the potential for biomolecular design at the atom level using language models.