# Characterization of anomalous diffusion through convolutional transformers

arXiv: https://arxiv.org/abs/2210.04959 · HTML: https://mlsanity.com/paper/2210.04959/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 46 references that are in the map.
- First posted on arXiv 10 Oct 2022.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Don't Decay the Learning Rate, Increase the Batch Size](https://mlsanity.com/paper/1711.00489/) (2017)
- [CMT: Convolutional Neural Networks Meet Vision Transformers](https://mlsanity.com/paper/2107.06263/) (2021)
- [ConvTransformer: A Convolutional Transformer Network for Video Frame Synthesis](https://mlsanity.com/paper/2011.10185/) (2020)
- [Efficient recurrent neural network methods for anomalously diffusing single particle short and noisy trajectories](https://mlsanity.com/paper/2108.02834/) (2021)
- [WaveNet-Based Deep Neural Networks for the Characterization of Anomalous Diffusion (WADNet)](https://mlsanity.com/paper/2106.08887/) (2021)

## Built on by (most cited first, 6 of 6)

- [Preface: Characterisation of Physical Processes from Anomalous Diffusion Data](https://mlsanity.com/paper/2301.00800/) (2023)
- [Gramian Angular Fields for leveraging pretrained computer vision models with anomalous diffusion trajectories](https://mlsanity.com/paper/2310.01416/) (2023)
- [Machine Learning Analysis of Anomalous Diffusion](https://mlsanity.com/paper/2412.01393/) (2024)
- [Machine-Learning Solutions for the Analysis of Single-Particle Diffusion Trajectories](https://mlsanity.com/paper/2308.09414/) (2023)
- [AnomalousNet: A Hybrid Approach with Attention U-Nets and Change Point Detection for Accurate Characterization of Anomalous Diffusion in Video Data](https://mlsanity.com/paper/2504.05271/) (2025)
- [Data-Efficient Learning of Anomalous Diffusion with Wavelet Representations: Enabling Direct Learning from Experimental Trajectories](https://mlsanity.com/paper/2512.08510/) (2025)

## Abstract (arXiv)

The results of the Anomalous Diffusion Challenge (AnDi Challenge) have shown that machine learning methods can outperform classical statistical methodology at the characterization of anomalous diffusion in both the inference of the anomalous diffusion exponent alpha associated with each trajectory (Task 1), and the determination of the underlying diffusive regime which produced such trajectories (Task 2). Furthermore, of the five teams that finished in the top three across both tasks of the AnDi challenge, three of those teams used recurrent neural networks (RNNs). While RNNs, like the long short-term memory (LSTM) network, are effective at learning long-term dependencies in sequential data, their key disadvantage is that they must be trained sequentially. In order to facilitate training with larger data sets, by training in parallel, we propose a new transformer based neural network architecture for the characterization of anomalous diffusion. Our new architecture, the Convolutional Transformer (ConvTransformer) uses a bi-layered convolutional neural network to extract features from our diffusive trajectories that can be thought of as being words in a sentence. These features are then fed to two transformer encoding blocks that perform either regression or classification. To our knowledge, this is the first time transformers have been used for characterizing anomalous diffusion. Moreover, this may be the first time that a transformer encoding block has been used with a convolutional neural network and without the need for a transformer decoding block or positional encoding. Apart from being able to train in parallel, we show that the ConvTransformer is able to outperform the previous state of the art at determining the underlying diffusive regime in short trajectories (length 10-50 steps), which are the most important for experimental researchers.