# Evaluation of Seismic Artificial Intelligence with Uncertainty

arXiv: https://arxiv.org/abs/2501.14809 · HTML: https://mlsanity.com/paper/2501.14809/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 54 references that are in the map.
- First posted on arXiv 15 Jan 2025.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Snapshot Ensembles: Train 1, get M for free](https://mlsanity.com/paper/1704.00109/) (2017)
- [Ensemble deep learning: A review](https://mlsanity.com/paper/2104.02395/) (2021)
- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [The Benchmark Lottery](https://mlsanity.com/paper/2107.07002/) (2021)
- [Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging](https://mlsanity.com/paper/1707.09861/) (2017)
- [SeisCLIP: A seismology foundation model pre-trained by multi-modal data for multi-purpose seismic feature extraction](https://mlsanity.com/paper/2309.02320/) (2023)
- [SeisLM: a Foundation Model for Seismic Waveforms](https://mlsanity.com/paper/2410.15765/) (2024)
- [Deep learning for seismic phase detection and picking in the aftershock zone of 2008 Mw7.9 Wenchuan earthquake](https://mlsanity.com/paper/1901.06396/) (2019)
- [Measuring the Data Efficiency of Deep Learning Methods](https://mlsanity.com/paper/1907.02549/) (2019)
- [SpanSeq: Similarity-based sequence data splitting method for improved development and assessment of deep learning projects](https://mlsanity.com/paper/2402.14482/) (2024)
- [A foundation model enpowered by a multi-modal prompt engine for universal seismic geobody interpretation across surveys](https://mlsanity.com/paper/2409.04962/) (2024)
- [Seismic-phase detection using multiple deep learning models for global and local representations of waveforms](https://mlsanity.com/paper/2211.02261/) (2022)

## Built on by (most cited first, 3 of 3)

- [The Trust Calibration Maturity Model for Characterizing and Communicating Trustworthiness of AI Systems](https://mlsanity.com/paper/2503.15511/) (2025)
- [Explainable AI for microseismic event detection](https://mlsanity.com/paper/2510.17458/) (2025)
- [Physics-Aware Machine Learning for Seismic and Volcanic Signal Interpretation](https://mlsanity.com/paper/2603.17855/) (2026)

## Abstract (arXiv)

Artificial intelligence has transformed the seismic community with deep learning models (DLMs) that are trained to complete specific tasks within workflows. However, there is still lack of robust evaluation frameworks for evaluating and comparing DLMs. We address this gap by designing an evaluation framework that jointly incorporates two crucial aspects: performance uncertainty and learning efficiency. To target these aspects, we meticulously construct the training, validation, and test splits using a clustering method tailored to seismic data and enact an expansive training design to segregate performance uncertainty arising from stochastic training processes and random data sampling. The framework's ability to guard against misleading declarations of model superiority is demonstrated through evaluation of PhaseNet [1], a popular seismic phase picking DLM, under 3 training approaches. Our framework helps practitioners choose the best model for their problem and set performance expectations by explicitly analyzing model performance with uncertainty at varying budgets of training data.