mlsanityData to 14 Oct 2026

Circuits, Features, and Heuristics in Molecular Transformers

arXiv 2512.09757 ↗First posted 10 Dec 2025cs.LG cs.AIcited by 2

On the map: Mechanistic interpretability → Interpreting biomedical and science models

Abstract (arXiv)

Transformers generate valid and diverse chemical structures, but little is known about the mechanisms that enable these models to capture the rules of molecular representation. We present a mechanistic analysis of autoregressive transformers trained on drug-like small molecules to reveal the computational structure underlying their capabilities across multiple levels of abstraction. We identify computational patterns consistent with low-level syntactic parsing and more abstract chemical validity constraints. Using sparse autoencoders (SAEs), we extract feature dictionaries associated with chemically relevant activation patterns. We validate our findings on downstream tasks and find that mechanistic insights can translate to predictive performance in various practical settings.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Interpreting biomedical and science models, one of 13 directions in Mechanistic interpretability. Builds on 32 of its 54 references that are in the map. First posted on arXiv 10 Dec 2025.

Builds on

32 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Interpreting and Steering Protein Language Models through Sparse Autoencoders13 Feb
  2. 2024InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoders13 Nov

Other references · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  5. 2020GLU Variants Improve Transformercited by 1,804
  6. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  8. 2019Attention is not Explanationcited by 929
  9. 2019Are Sixteen Heads Really Better than One?cited by 816
  10. 2019GNNExplainer: Generating Explanations for Graph Neural Networkscited by 737
  11. 2019Attention is not not Explanationcited by 636
  12. 2024Scaling and evaluating sparse autoencoderscited by 553
18 more
  1. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  2. 2019Analyzing Learned Molecular Representations for Property Predictioncited by 375
  3. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 337
  4. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  5. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 288
  6. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 264
  7. 2018Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Predictioncited by 229
  8. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 206
  9. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  10. 2021Thinking Like Transformerscited by 184
  11. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 179
  12. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
  13. 2020Molecular representation learning with language models and domain-relevant auxiliary taskscited by 94
  14. 2025Sparse Autoencoders Trained on the Same Data Learn Different Featurescited by 93
  15. 2019SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discoverycited by 90
  16. 2017SMILES Enumeration as Data Augmentation for Neural Network Modeling of Moleculescited by 59
  17. 2020How Can Self-Attention Networks Recognize Dyck-n Languages?cited by 51
  18. 2023ADMET property prediction through combinations of molecular fingerprintscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SMolLM: Small Language Models Learn Small Molecular Grammarcited by 0
  2. 2026From Syntax to Semantics: Unveiling the Emergence of Chirality in SMILES Translation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Molecules Meet Language: Confound-Aware Representation Learning and Chemical Property Steering in Transformer-VAE Latent Spaces
  2. 2026What Does a Chemical Language Model Know About Molecules?
  3. 2025Unveiling Latent Knowledge in Chemistry Language Models through Sparse Autoencoders
  4. 2025Atom-anchored LLMs speak Chemistry: A Retrosynthesis Demonstration
  5. 2026PairIT: Autoregressive Transformers for Low-Data Molecule Optimization
  6. 2026Agentic generation of verifiable rules for deterministic, self-expanding reaction classification
  7. 2025ATOM: A Pretrained Neural Operator for Multitask Molecular Dynamics
  8. 2026MoleCode unlocks structural intelligence in large language models