# What Does a Chemical Language Model Know About Molecules?

arXiv: https://arxiv.org/abs/2606.23443 · HTML: https://mlsanity.com/paper/2606.23443/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 37 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Builds on (most cited first, 13 of 13)

- [Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development](https://mlsanity.com/paper/2102.09548/) (2021)
- [Large-Scale Chemical Language Representations Capture Molecular Structure and Properties](https://mlsanity.com/paper/2106.09553/) (2021)
- [Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet](https://mlsanity.com/paper/2605.29358/) (2026)
- [ChemBERTa-2: Towards Chemical Foundation Models](https://mlsanity.com/paper/2209.01712/) (2022)
- [InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2412.12101/) (2024)
- [SMILES Enumeration as Data Augmentation for Neural Network Modeling of Molecules](https://mlsanity.com/paper/1703.07076/) (2017)
- [polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informatics](https://mlsanity.com/paper/2209.14803/) (2022)
- [LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures](https://mlsanity.com/paper/2509.14252/) (2025)
- [Into the Rabbit Hull: From Task-Relevant Concepts in DINO to Minkowski Geometry](https://mlsanity.com/paper/2510.08638/) (2025)
- [Dense SAE Latents Are Features, Not Bugs](https://mlsanity.com/paper/2506.15679/) (2025)
- [Protein Circuit Tracing via Cross-layer Transcoders](https://mlsanity.com/paper/2602.12026/) (2026)
- [NPGPT: Natural Product-Like Compound Generation with GPT-based Chemical Language Models](https://mlsanity.com/paper/2411.12886/) (2024)
- [Unveiling Latent Knowledge in Chemistry Language Models through Sparse Autoencoders](https://mlsanity.com/paper/2512.08077/) (2025)

## Built on by (most cited first, 2 of 2)

- [Shared Geometry Is Not Shared Physics: A Layerwise Test of the Platonic Representation Hypothesis in Astronomy](https://mlsanity.com/paper/2610.04130/) (2026)
- [Does Pretraining Teach Chemical Language Models Chemistry?](https://mlsanity.com/paper/2610.18651/) (2026)

## Abstract (arXiv)

Chemical language models (cLMs) are widely assumed to learn surface-level syntactic patterns rather than learning meaningful molecular semantics. Here, we apply sparse autoencoders (SAEs) to MolFormer, an encoder-only cLM, to mechanistically examine how molecular representations are built across layers. We discover that early layers rely on position-tracking latents to parse molecular grammar, while later layers encode atom-in-substructure and pharmacologically relevant features. Additionally, we show that non-canonical SMILES produce more disruptive representation shifts than invalid SMILES, driven by position-latent disruption propagating across layers. To support further exploration, we develop InterMol, an interactive visualizer for SAE activations on molecular strings and structures.