# Transformers for scientific data: a pedagogical review for astronomers

arXiv: https://arxiv.org/abs/2310.12069 · HTML: https://mlsanity.com/paper/2310.12069/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 22 references that are in the map.
- First posted on arXiv 18 Oct 2023.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Astroformer: More Data Might not be all you need for Classification](https://mlsanity.com/paper/2304.05350/) (2023)
- [Deep Attention-Based Supernovae Classification of Multi-Band Light-Curves](https://mlsanity.com/paper/2201.08482/) (2022)
- [Don't Pay Attention to the Noise: Learning Self-supervised Representations of Light Curves with a Denoising Time Series Transformer](https://mlsanity.com/paper/2207.02777/) (2022)
- [Distinguishing a planetary transit from false positives: a Transformer-based classification for planetary transit signals](https://mlsanity.com/paper/2304.14283/) (2023)
- [Astroconformer: Inferring Surface Gravity of Stars from Stellar Light Curves with Transformer](https://mlsanity.com/paper/2207.02787/) (2022)
- [Detection of Strongly Lensed Arcs in Galaxy Clusters with Transformers](https://mlsanity.com/paper/2211.05972/) (2022)

## Built on by (most cited first, 2 of 2)

- [Vision Transformer for Transient Noise Classification](https://mlsanity.com/paper/2510.06273/) (2025)
- [Leveraging Physiological Signals to Predict Exam Outcomes with Machine Learning](https://mlsanity.com/paper/2606.14960/) (2026)

## Abstract (arXiv)

The deep learning architecture associated with ChatGPT and related generative AI products is known as transformers. Initially applied to Natural Language Processing, transformers and the self-attention mechanism they exploit have gained widespread interest across the natural sciences. The goal of this pedagogical and informal review is to introduce transformers to scientists. The review includes the mathematics underlying the attention mechanism, a description of the original transformer architecture, and a section on applications to time series and imaging data in astronomy. We include a Frequently Asked Questions section for readers who are curious about generative AI or interested in getting started with transformers for their research problem.