mlsanityData to 7 Oct 2026

SMolLM: Small Language Models Learn Small Molecular Grammar

arXiv 2605.06322 ↗First posted 7 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Language models for molecular design have scaled to hundreds of millions of parameters, yet how they learn chemical grammar is poorly understood. We train SMolLM, a 53K-parameter weight-shared transformer, to generate novel SMILES with 95% validity on the ZINC-250K drug-like-molecule benchmark, outperforming a standard GPT with 10 times more parameters. Mechanistically, the same block resolves SMILES constraints across passes in a fixed hierarchy: brackets first, rings second, and valence last, as shown by error classification and linear probing, with ablation isolating the bracket-matching head. Together, these results yield a compact, mechanistically interpretable molecular generator and a testbed for studying iterative computation in formal-language domains.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 41 of its 53 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

41 of 53 references.

References · 41

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  4. 2020GLU Variants Improve Transformercited by 1,724
  5. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  6. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  7. 2022In-context Learning and Induction Headscited by 930
  8. 2022Transformers learn in-context by gradient descentcited by 714
  9. 2018Universal Transformerscited by 613
  10. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  11. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  12. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
29 more
  1. 2017Molecular De Novo Design through Deep Reinforcement Learningcited by 264
  2. 2023Looped Transformers as Programmable Computerscited by 224
  3. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  4. 2022Neural Networks and the Chomsky Hierarchycited by 214
  5. 2018Fr'echet ChemNet Distance: A metric for generative models for molecules in drug discoverycited by 191
  6. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  7. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  8. 2017SMILES Enumeration as Data Augmentation for Neural Network Modeling of Moleculescited by 58
  9. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 45
  10. 2021Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformerscited by 42
  11. 2025Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biasescited by 36
  12. 2026Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformerscited by 34
  13. 2023Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammarscited by 30
  14. 2025Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformerscited by 30
  15. 2026A Mechanistic Analysis of Looped Reasoning Language Modelscited by 27
  16. 2025Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformercited by 24
  17. 2025TIDE : Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image Generationcited by 16
  18. 2024GP-MoLFormer: A Foundation Model For Molecular Generationcited by 11
  19. 2026ELT: Elastic Looped Transformers for Visual Generationcited by 11
  20. 2026Hyperloop Transformerscited by 11
  21. 2026Dynamics Within Latent Chain-of-Thought: An Empirical Study of Causal Structurecited by 9
  22. 2026SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursioncited by 9
  23. 2026Are Latent Reasoning Models Easily Interpretable?cited by 6
  24. 2026Step-resolved data attribution for looped transformerscited by 5
  25. 2025Unveiling Latent Knowledge in Chemistry Language Models through Sparse Autoencoderscited by 2
  26. 2025Circuits, Features, and Heuristics in Molecular Transformerscited by 2
  27. 2026Energy-Entropy Regularization: The True Power of Minimal Looped Transformerscited by 2
  28. 2022Chemical transformer compression for accelerating both training and inference of molecular modelingcited by 1
  29. 2026Unveiling Scaling Behaviors in Molecular Language Models: Effects of Model Size, Data, and Representationcited by 1

Built on by

Not cited yet.