# PEMS: Pre-trained Epidemic Time-series Models

arXiv: https://arxiv.org/abs/2311.07841 · HTML: https://mlsanity.com/paper/2311.07841/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 60 references that are in the map.
- First posted on arXiv 14 Nov 2023.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis](https://mlsanity.com/paper/2210.02186/) (2022)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [AutoFormer: Searching Transformers for Visual Recognition](https://mlsanity.com/paper/2107.00651/) (2021)
- [Structured Inference Networks for Nonlinear State Space Models](https://mlsanity.com/paper/1609.09869/) (2016)
- [A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data](https://mlsanity.com/paper/1811.08055/) (2018)
- [A Survey of Vision-Language Pre-Trained Models](https://mlsanity.com/paper/2202.10936/) (2022)
- [The Functional Neural Process](https://mlsanity.com/paper/1906.08324/) (2019)
- [Learning Interpretable Deep State Space Model for Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2102.00397/) (2021)
- [Bridging Physics-based and Data-driven modeling for Learning Dynamical Systems](https://mlsanity.com/paper/2011.10616/) (2020)
- [EINNs: Epidemiologically-informed Neural Networks](https://mlsanity.com/paper/2202.10446/) (2022)
- [Accurate estimation of influenza epidemics using Google search data via ARGO](https://mlsanity.com/paper/1505.00864/) (2015)
- [CAMul: Calibrated and Accurate Multi-view Time-Series Forecasting](https://mlsanity.com/paper/2109.07438/) (2021)

## Built on by (most cited first, 3 of 3)

- [Position: What Can Large Language Models Tell Us about Time Series Analysis](https://mlsanity.com/paper/2402.02713/) (2024)
- [Pre-training Epidemic Time Series Forecasters with Compartmental Prototypes](https://mlsanity.com/paper/2502.03393/) (2025)
- [IDOBE: Infectious Disease Outbreak forecasting Benchmark Ecosystem](https://mlsanity.com/paper/2604.18521/) (2026)

## Abstract (arXiv)

Providing accurate and reliable predictions about the future of an epidemic is an important problem for enabling informed public health decisions. Recent works have shown that leveraging data-driven solutions that utilize advances in deep learning methods to learn from past data of an epidemic often outperform traditional mechanistic models. However, in many cases, the past data is sparse and may not sufficiently capture the underlying dynamics. While there exists a large amount of data from past epidemics, leveraging prior knowledge from time-series data of other diseases is a non-trivial challenge. Motivated by the success of pre-trained models in language and vision tasks, we tackle the problem of pre-training epidemic time-series models to learn from multiple datasets from different diseases and epidemics. We introduce Pre-trained Epidemic Time-Series Models (PEMS) that learn from diverse time-series datasets of a variety of diseases by formulating pre-training as a set of self-supervised learning (SSL) tasks. We tackle various important challenges specific to pre-training for epidemic time-series such as dealing with heterogeneous dynamics and efficiently capturing useful patterns from multiple epidemic datasets by carefully designing the SSL tasks to learn important priors about the epidemic dynamics that can be leveraged for fine-tuning to multiple downstream tasks. The resultant PEM outperforms previous state-of-the-art methods in various downstream time-series tasks across datasets of varying seasonal patterns, geography, and mechanism of contagion including the novel Covid-19 pandemic unseen in pre-trained data with better efficiency using smaller fraction of datasets.