mlsanityData to 14 Oct 2026

What Does a Chemical Language Model Know About Molecules?

arXiv 2606.23443 ↗First posted 22 Jun 2026cs.LG cs.AI physics.chem-phcited by 2

Not on the map. Why?

Abstract (arXiv)

Chemical language models (cLMs) are widely assumed to learn surface-level syntactic patterns rather than learning meaningful molecular semantics. Here, we apply sparse autoencoders (SAEs) to MolFormer, an encoder-only cLM, to mechanistically examine how molecular representations are built across layers. We discover that early layers rely on position-tracking latents to parse molecular grammar, while later layers encode atom-in-substructure and pharmacologically relevant features. Additionally, we show that non-canonical SMILES produce more disruptive representation shifts than invalid SMILES, driven by position-latent disruption propagating across layers. To support further exploration, we develop InterMol, an interactive visualizer for SAE activations on molecular strings and structures.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 13 of its 37 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

13 of 37 references.

References · 13

  1. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 264
  2. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 206
  3. 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 191
  4. 2022ChemBERTa-2: Towards Chemical Foundation Modelscited by 127
  5. 2024InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoderscited by 66
  6. 2017SMILES Enumeration as Data Augmentation for Neural Network Modeling of Moleculescited by 59
  7. 2022polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informaticscited by 48
  8. 2025LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architecturescited by 38
  9. 2025Into the Rabbit Hull: From Task-Relevant Concepts in DINO to Minkowski Geometrycited by 34
  10. 2025Dense SAE Latents Are Features, Not Bugscited by 17
  11. 2026Protein Circuit Tracing via Cross-layer Transcoderscited by 3
  12. 2024NPGPT: Natural Product-Like Compound Generation with GPT-based Chemical Language Modelscited by 2
1 more
  1. 2025Unveiling Latent Knowledge in Chemistry Language Models through Sparse Autoencoderscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Shared Geometry Is Not Shared Physics: A Layerwise Test of the Platonic Representation Hypothesis in Astronomycited by 0
  2. 2026Does Pretraining Teach Chemical Language Models Chemistry?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Omni-Mol: Multitask Molecular Model for Any-to-any Modalities
  2. 2025Thinking like a CHEMIST: Combined Heterogeneous Embedding Model Integrating Structure and Tokens
  3. 2026Rethinking Molecular Text Representations for LLMs: An Empirical Study
  4. 2026MolEmb: Multimodal Large Language Models Can Be Strong Molecular Embedding Models
  5. 2025NovoMolGen: Rethinking Molecular Language Model Pretraining
  6. 2026Molecules Meet Language: Confound-Aware Representation Learning and Chemical Property Steering in Transformer-VAE Latent Spaces
  7. 2026MoleCode unlocks structural intelligence in large language models
  8. 2025Circuits, Features, and Heuristics in Molecular Transformers