STEB: In Search of the Best Evaluation Approach for Synthetic Time Series
Not on the map. Why?
Abstract (arXiv)
The growing need for synthetic time series, due to data augmentation or privacy regulations, has led to numerous generative models, frameworks, and evaluation measures alike. Objectively comparing these measures on a large scale remains an open challenge. We propose the Synthetic Time series Evaluation Benchmark (STEB) -- the first benchmark framework that enables comprehensive and interpretable automated comparisons of synthetic time series evaluation measures. Using 10 diverse datasets, randomness injection, and 13 configurable data transformations, STEB computes indicators for measure reliability and score consistency. It tracks running time, test errors, and features sequential and parallel modes of operation. In our experiments, we determine a ranking of 41 measures from literature and confirm that the choice of upstream time series embedding heavily impacts the final score.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 31 of its 58 references that are in the map. First posted on arXiv 27 May 2025.
Builds on
31 of 58 references.
References · 31
- 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
- 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 879
- 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 434
- 2017Real-valued (Medical) Time Series Generation with Recurrent Conditional GANscited by 333
- 2015A note on the evaluation of generative modelscited by 292
- 2019Unsupervised Scalable Representation Learning for Multivariate Time Seriescited by 269
- 2016Revisiting Classifier Two-Sample Testscited by 268
- 2017Are GANs Created Equal? A Large-Scale Studycited by 267
- 2021How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Modelscited by 196
- 2019Classification Accuracy Score for Conditional Generative Modelscited by 136
- 2021DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networkscited by 109
- 2018An empirical study on evaluation metrics of generative adversarial networkscited by 81
19 more
- 2020COT-GAN: Generating Sequential Data via Causal Optimal Transportcited by 71
- 2023Membership Inference Attacks against Synthetic Data through Overfitting Detectioncited by 66
- 2018On GANs and GMMscited by 62
- 2021PSA-GAN: Progressive Self Attention GANs for Synthetic Time Seriescited by 58
- 2016UniMiB SHAR: a new dataset for human activity recognition using acceleration data from smartphonescited by 54
- 2020A Non-Parametric Test to Detect Data-Copying in Generative Modelscited by 52
- 2022TTS-GAN: A Transformer-based Time-Series Generative Adversarial Networkcited by 41
- 2021Manifold Topology Divergence: a Framework for Comparing Data Manifoldscited by 31
- 2023Beyond Privacy: Navigating the Opportunities and Challenges of Synthetic Datacited by 29
- 2023TSGBench: Time Series Generation Benchmarkcited by 26
- 2019Deep Hedging: Learning to Simulate Equity Option Marketscited by 25
- 2022Generating multivariate time series with COmmon Source CoordInated GAN (COSCI-GAN)cited by 23
- 2022TTS-CGAN: A Transformer Time-Series Conditional GAN for Biosignal Data Augmentationcited by 21
- 2023TSGM: A Flexible Framework for Generative Modeling of Synthetic Time Seriescited by 18
- 2023PCF-GAN: generating sequential data via the characteristic function of measures on the path spacecited by 15
- 2021On the Fairness of Generative Adversarial Networks (GANs)cited by 14
- 2024Establishing a Unified Evaluation Framework for Human Motion Generation: A Comparative Analysis of Metricscited by 4
- 2021On Characterizing GAN Convergence Through Proximal Duality Gapcited by 3
- 2019Visual Evaluation of Generative Adversarial Networks for Time Series Datacited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.