# STEB: In Search of the Best Evaluation Approach for Synthetic Time Series

arXiv: https://arxiv.org/abs/2505.21160 · HTML: https://mlsanity.com/paper/2505.21160/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 58 references that are in the map.
- First posted on arXiv 27 May 2025.

## Builds on (most cited first, 30 of 31)

- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Reliable Fidelity and Diversity Metrics for Generative Models](https://mlsanity.com/paper/2002.09797/) (2020)
- [Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs](https://mlsanity.com/paper/1706.02633/) (2017)
- [A note on the evaluation of generative models](https://mlsanity.com/paper/1511.01844/) (2015)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Revisiting Classifier Two-Sample Tests](https://mlsanity.com/paper/1610.06545/) (2016)
- [Are GANs Created Equal? A Large-Scale Study](https://mlsanity.com/paper/1711.10337/) (2017)
- [How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models](https://mlsanity.com/paper/2102.08921/) (2021)
- [Classification Accuracy Score for Conditional Generative Models](https://mlsanity.com/paper/1905.10887/) (2019)
- [DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networks](https://mlsanity.com/paper/2110.12884/) (2021)
- [An empirical study on evaluation metrics of generative adversarial networks](https://mlsanity.com/paper/1806.07755/) (2018)
- [COT-GAN: Generating Sequential Data via Causal Optimal Transport](https://mlsanity.com/paper/2006.08571/) (2020)
- [Membership Inference Attacks against Synthetic Data through Overfitting Detection](https://mlsanity.com/paper/2302.12580/) (2023)
- [On GANs and GMMs](https://mlsanity.com/paper/1805.12462/) (2018)
- [PSA-GAN: Progressive Self Attention GANs for Synthetic Time Series](https://mlsanity.com/paper/2108.00981/) (2021)
- [UniMiB SHAR: a new dataset for human activity recognition using acceleration data from smartphones](https://mlsanity.com/paper/1611.07688/) (2016)
- [A Non-Parametric Test to Detect Data-Copying in Generative Models](https://mlsanity.com/paper/2004.05675/) (2020)
- [TTS-GAN: A Transformer-based Time-Series Generative Adversarial Network](https://mlsanity.com/paper/2202.02691/) (2022)
- [Manifold Topology Divergence: a Framework for Comparing Data Manifolds](https://mlsanity.com/paper/2106.04024/) (2021)
- [Beyond Privacy: Navigating the Opportunities and Challenges of Synthetic Data](https://mlsanity.com/paper/2304.03722/) (2023)
- [TSGBench: Time Series Generation Benchmark](https://mlsanity.com/paper/2309.03755/) (2023)
- [Deep Hedging: Learning to Simulate Equity Option Markets](https://mlsanity.com/paper/1911.01700/) (2019)
- [Generating multivariate time series with COmmon Source CoordInated GAN (COSCI-GAN)](https://mlsanity.com/paper/2205.13741/) (2022)
- [TTS-CGAN: A Transformer Time-Series Conditional GAN for Biosignal Data Augmentation](https://mlsanity.com/paper/2206.13676/) (2022)
- [TSGM: A Flexible Framework for Generative Modeling of Synthetic Time Series](https://mlsanity.com/paper/2305.11567/) (2023)
- [PCF-GAN: generating sequential data via the characteristic function of measures on the path space](https://mlsanity.com/paper/2305.12511/) (2023)
- [On the Fairness of Generative Adversarial Networks (GANs)](https://mlsanity.com/paper/2103.00950/) (2021)
- [Establishing a Unified Evaluation Framework for Human Motion Generation: A Comparative Analysis of Metrics](https://mlsanity.com/paper/2405.07680/) (2024)
- [On Characterizing GAN Convergence Through Proximal Duality Gap](https://mlsanity.com/paper/2105.04801/) (2021)

## Built on by (most cited first, 1 of 1)

- [HyperNSDE: Personalized Neural SDEs for Joint Static-Longitudinal Clinical Data Generation](https://mlsanity.com/paper/2610.07383/) (2026)

## Abstract (arXiv)

The growing need for synthetic time series, due to data augmentation or privacy regulations, has led to numerous generative models, frameworks, and evaluation measures alike. Objectively comparing these measures on a large scale remains an open challenge. We propose the Synthetic Time series Evaluation Benchmark (STEB) -- the first benchmark framework that enables comprehensive and interpretable automated comparisons of synthetic time series evaluation measures. Using 10 diverse datasets, randomness injection, and 13 configurable data transformations, STEB computes indicators for measure reliability and score consistency. It tracks running time, test errors, and features sequential and parallel modes of operation. In our experiments, we determine a ranking of 41 measures from literature and confirm that the choice of upstream time series embedding heavily impacts the final score.