# SeisLM: a Foundation Model for Seismic Waveforms

arXiv: https://arxiv.org/abs/2410.15765 · HTML: https://mlsanity.com/paper/2410.15765/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 73 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Finite Scalar Quantization: VQ-VAE Made Simple](https://mlsanity.com/paper/2309.15505/) (2023)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Adaptive Input Representations for Neural Language Modeling](https://mlsanity.com/paper/1809.10853/) (2018)
- [Transformers without Tears: Improving the Normalization of Self-Attention](https://mlsanity.com/paper/1910.05895/) (2019)
- [The challenge of realistic music generation: modelling raw audio at scale](https://mlsanity.com/paper/1806.10474/) (2018)
- [Robust Training of Vector Quantized Bottleneck Models](https://mlsanity.com/paper/2005.08520/) (2020)
- [CRED: A Deep Residual Network of Convolutional and Recurrent Units for Earthquake Signal Detection](https://mlsanity.com/paper/1810.01965/) (2018)
- [SeisCLIP: A seismology foundation model pre-trained by multi-modal data for multi-purpose seismic feature extraction](https://mlsanity.com/paper/2309.02320/) (2023)
- [SeisT: A foundational deep learning model for earthquake monitoring tasks](https://mlsanity.com/paper/2310.01037/) (2023)
- [End-to-end LSTM based estimation of volcano event epicenter localization](https://mlsanity.com/paper/2110.14594/) (2021)

## Built on by (most cited first, 8 of 8)

- [Evaluation of Seismic Artificial Intelligence with Uncertainty](https://mlsanity.com/paper/2501.14809/) (2025)
- [SeisMoLLM: Advancing Seismic Monitoring via Cross-modal Transfer with Pre-trained Large Language Model](https://mlsanity.com/paper/2502.19960/) (2025)
- [AutoCas: Autoregressive Cascade Predictor in Social Networks via Large Language Models](https://mlsanity.com/paper/2502.18040/) (2025)
- [Data-Driven Forecasting of three-Component Seismograms Using Transformer Architectures](https://mlsanity.com/paper/2606.02912/) (2026)
- [Physics-Aware Machine Learning for Seismic and Volcanic Signal Interpretation](https://mlsanity.com/paper/2603.17855/) (2026)
- [GNSS-FM: A Self-Supervised Foundation Model for Daily GNSS Displacement Time Series](https://mlsanity.com/paper/2606.07725/) (2026)
- [IQ-JEPA: A Joint-Embedding Predictive Architecture with a Hermitian Vision Transformer for Sound Speed and Attenuation Estimation from Ultrasound IQ Data](https://mlsanity.com/paper/2607.22351/) (2026)
- [TomoTransformer: Towards a Foundation Model for CT Reconstruction](https://mlsanity.com/paper/2609.37605/) (2026)

## Abstract (arXiv)

We introduce the Seismic Language Model (SeisLM), a foundational model designed to analyze seismic waveforms -- signals generated by Earth's vibrations such as the ones originating from earthquakes. SeisLM is pretrained on a large collection of open-source seismic datasets using a self-supervised contrastive loss, akin to BERT in language modeling. This approach allows the model to learn general seismic waveform patterns from unlabeled data without being tied to specific downstream tasks. When fine-tuned, SeisLM excels in seismological tasks like event detection, phase-picking, onset time regression, and foreshock-aftershock classification. The code has been made publicly available on https://github.com/liutianlin0121/seisLM.