# Integrating Transformer and Autoencoder Techniques with Spectral Graph Algorithms for the Prediction of Scarcely Labeled Molecular Data

arXiv: https://arxiv.org/abs/2211.06759 · HTML: https://mlsanity.com/paper/2211.06759/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 71 references that are in the map.
- First posted on arXiv 12 Nov 2022.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Unsupervised Domain Adaptation with Residual Transfer Networks](https://mlsanity.com/paper/1602.04433/) (2016)
- [Partial Transfer Learning with Selective Adversarial Networks](https://mlsanity.com/paper/1707.07901/) (2017)
- [Deep Active Learning over the Long Tail](https://mlsanity.com/paper/1711.00941/) (2017)
- [Deep Active Learning with a Neural Architecture Search](https://mlsanity.com/paper/1811.07579/) (2018)
- [Multiclass Data Segmentation using Diffuse Interface Methods on Graphs](https://mlsanity.com/paper/1302.3913/) (2013)
- [Multiscale Laplacian Learning](https://mlsanity.com/paper/2109.03718/) (2021)
- [Proteome-informed machine learning studies of cocaine addiction](https://mlsanity.com/paper/2109.08718/) (2021)

## Built on by (most cited first, 2 of 2)

- [Persistent Laplacian-enhanced Algorithm for Scarcely Labeled Data Classification](https://mlsanity.com/paper/2305.16239/) (2023)
- [Graph-Based Bidirectional Transformer Decision Threshold Adjustment Algorithm for Class-Imbalanced Molecular Data](https://mlsanity.com/paper/2406.06479/) (2024)

## Abstract (arXiv)

In molecular and biological sciences, experiments are expensive, time-consuming, and often subject to ethical constraints. Consequently, one often faces the challenging task of predicting desirable properties from small data sets or scarcely-labeled data sets. Although transfer learning can be advantageous, it requires the existence of a related large data set. This work introduces three graph-based models incorporating Merriman-Bence-Osher (MBO) techniques to tackle this challenge. Specifically, graph-based modifications of the MBO scheme are integrated with state-of-the-art techniques, including a home-made transformer and an autoencoder, in order to deal with scarcely-labeled data sets. In addition, a consensus technique is detailed. The proposed models are validated using five benchmark data sets. We also provide a thorough comparison to other competing methods, such as support vector machines, random forests, and gradient boosting decision trees, which are known for their good performance on small data sets. The performances of various methods are analyzed using residue-similarity (R-S) scores and R-S indices. Extensive computational experiments and theoretical analysis show that the new models perform very well even when as little as 1% of the data set is used as labeled data.