mlsanityData to 14 Oct 2026

Superior Molecular Representations from Intermediate Encoder Layers

arXiv 2506.06443 ↗First posted 6 Jun 2025cs.LG cs.AI physics.chem-phcited by 4

Not on the map. Why?

Abstract (arXiv)

Pretrained molecular encoders have become indispensable in computational chemistry for tasks such as property prediction and molecular generation. However, the standard practice of relying solely on final-layer embeddings for downstream tasks may discard valuable information. In this work, we first analyze the information flow in five diverse molecular encoders and find that intermediate layers retain more general-purpose features, whereas the final-layer specializes and compresses information. We then perform an empirical layer-wise evaluation across 22 property prediction tasks. We find that using frozen embeddings from optimal intermediate layers improves downstream performance by an average of 5.4%, up to 28.6%, compared to the final-layer. Furthermore, finetuning encoders truncated at intermediate depths achieves even greater average improvements of 8.5%, with increases as high as 40.8%, obtaining new state-of-the-art results on several benchmarks. These findings highlight the importance of exploring the full representational depth of molecular encoders to achieve substantial performance improvements and computational efficiency. The code will be made publicly available.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 23 of its 55 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

23 of 55 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  3. 2017Inductive Representation Learning on Large Graphscited by 6,320
  4. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  5. 2018How Powerful are Graph Neural Networks?cited by 4,393
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  7. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  8. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  9. 2022MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fieldscited by 474
  10. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  11. 2019Analyzing Learned Molecular Representations for Property Predictioncited by 375
  12. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 337
11 more
  1. 2023Language Models Represent Space and Timecited by 316
  2. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  3. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 264
  4. 2018CatBoost: gradient boosting with categorical features supportcited by 208
  5. 2020Machine Learning Force Fieldscited by 207
  6. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 206
  7. 2023The geometry of hidden representations of large transformer modelscited by 112
  8. 2024On the Scalability of GNNs for Molecular Graphscited by 36
  9. 2017SMILES2Vec: An Interpretable General-Purpose Deep Neural Network for Predicting Chemical Propertiescited by 30
  10. 2023ADMET property prediction through combinations of molecular fingerprintscited by 11
  11. 2024Improving Performance Prediction of Electrolyte Formulations with Transformer-based Molecular Representation Modelcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Tabular foundation models for in-context prediction of molecular propertiescited by 6
  2. 2026TriForces: Augmenting Atomistic GNNs for Transferable Representationscited by 1
  3. 2026When Tabular Foundation Models Transfer Across Modalities: A Systematic Evaluation Across 95 Datasets, 7 Modalities, and Two Regimescited by 0
  4. 2026ElemeNet: Multiscale Molecular Machine Learning with Uncertainty Quantification Across the Periodic Tablecited by 0