mlsanityData to 7 Oct 2026

Explainability Techniques for Chemical Language Models

arXiv 2305.16192 ↗First posted 25 May 2023cs.LG cs.AI physics.chem-phcited by 6

Not on the map. Why?

Abstract (arXiv)

Explainability techniques are crucial in gaining insights into the reasons behind the predictions of deep learning models, which have not yet been applied to chemical language models. We propose an explainable AI technique that attributes the importance of individual atoms towards the predictions made by these models. Our method backpropagates the relevance information towards the chemical input string and visualizes the importance of individual atoms. We focus on self-attention Transformers operating on molecular string representations and leverage a pretrained encoder for finetuning. We showcase the method by predicting and visualizing solubility in water and organic solvents. We achieve competitive model performance while obtaining interpretable predictions, which we use to inspect the pretrained model.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 27 of its 57 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

27 of 57 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  8. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  9. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  10. 2020Quantifying Attention Flow in Transformerscited by 759
  11. 2015Convolutional Networks on Graphs for Learning Molecular Fingerprintscited by 751
  12. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
15 more
  1. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  2. 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 295
  3. 2015Explaining NonLinear Classification Decisions with Deep Taylor Decompositioncited by 248
  4. 2019Explainability Techniques for Graph Convolutional Networkscited by 160
  5. 2021On Calibration and Out-of-domain Generalizationcited by 134
  6. 2020Molecule Attention Transformercited by 92
  7. 2020Molecular representation learning with language models and domain-relevant auxiliary taskscited by 91
  8. 2020Mapping the Space of Chemical Reactions Using Attention-Based Neural Networkscited by 70
  9. 2022Formal Algorithms for Transformerscited by 70
  10. 2021Keeping it Simple: Language Models can learn Complex Molecular Distributionscited by 68
  11. 2022MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Designcited by 66
  12. 2020Deep Molecular Dreaming: Inverse machine learning for de-novo molecular design and interpretability with surjective representationscited by 20
  13. 2022Safe Reinforcement Learning From Pixels Using a Stochastic Latent Representationcited by 11
  14. 2023Recent advances in the Self-Referencing Embedding Strings (SELFIES) librarycited by 8
  15. 2022Neural Payoff Machines: Predicting Fair and Stable Payoff Allocations Among Team Memberscited by 6

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2; 2025: 2.

6 papers.

  1. 2024FragNet: A Graph Neural Network for Molecular Property Prediction with Four Levels of Interpretabilitycited by 3
  2. 2023Using Cooperative Game Theory to Prune Neural Networkscited by 1
  3. 2025Conditional Chemical Language Models are Versatile Tools in Drug Discoverycited by 1
  4. 2023Explainable Multi-View Deep Networks Methodology for Experimental Physicscited by 0
  5. 2024The Role of Model Architecture and Scale in Predicting Molecular Properties: Insights from Fine-Tuning RoBERTa, BART, and LLaMAcited by 0
  6. 2025Modelling Chemical Reaction Networks using Neural Ordinary Differential Equationscited by 0