# EvEMTBench: An Open Benchmark for Machine Learning in Power System Protection

arXiv: https://arxiv.org/abs/2609.28149 · HTML: https://mlsanity.com/paper/2609.28149/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 47 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Builds on (most cited first, 19 of 19)

- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [The Benchmark Lottery](https://mlsanity.com/paper/2107.07002/) (2021)
- [Why rankings of biomedical image analysis competitions should be interpreted with care](https://mlsanity.com/paper/1806.02051/) (2018)
- [Fault Location in Power Distribution Systems via Deep Graph Convolutional Networks](https://mlsanity.com/paper/1812.09464/) (2018)
- [PowerGraph: A power grid benchmark dataset for graph neural networks](https://mlsanity.com/paper/2402.02827/) (2024)
- [A Multi-scale Time-series Dataset with Benchmark for Machine Learning in Decarbonized Energy Grids](https://mlsanity.com/paper/2110.06324/) (2021)
- [A Scoping Review of Machine Learning Applications in Power System Protection and Disturbance Management](https://mlsanity.com/paper/2509.09053/) (2025)
- [Spatial-Temporal Recurrent Graph Neural Networks for Fault Diagnostics in Power Distribution Systems](https://mlsanity.com/paper/2210.15177/) (2022)
- [Impact of Data Sparsity on Machine Learning for Fault Detection in Power System Protection](https://mlsanity.com/paper/2505.15560/) (2025)
- [Robustness Evaluation of Machine Learning Models for Fault Classification and Localization In Power System Protection](https://mlsanity.com/paper/2512.15385/) (2025)
- [Controlled Comparison of Machine Learning Models for Fault Classification and Localization in Power System Protection](https://mlsanity.com/paper/2510.00831/) (2025)
- [PROTECT-90: A Fault Dataset for Power System Protection](https://mlsanity.com/paper/2606.24298/) (2026)
- [A Heterogeneous Graph-Based Multi-Task Learning for Fault Event Diagnosis in Smart Grid](https://mlsanity.com/paper/2309.09921/) (2023)
- [Latency-Aware Deep Learning Benchmark for Real-Time Cyber-Physical Attack and Fault Classification in Inverter-Dominated Power Grids](https://mlsanity.com/paper/2605.17256/) (2026)
- [A Systematic Evaluation of Machine Learning Methods for Fault Detection and Line Identification in Electrical Power Grids](https://mlsanity.com/paper/2609.16744/) (2026)
- [Fault Inception Detection in Real-World Disturbance Data for Power System Protection](https://mlsanity.com/paper/2606.23111/) (2026)
- [A simulation based dataset of faults and events for machine learning in power systems](https://mlsanity.com/paper/2608.19777/) (2026)
- [Unsupervised Clustering for Fault Analysis in High-Voltage Power Systems Using Voltage and Current Signals](https://mlsanity.com/paper/2505.17763/) (2025)
- [A Standardized Framework for Machine Learning in Power System Protection](https://mlsanity.com/paper/2608.20181/) (2026)

## Abstract (arXiv)

Studies of machine-learning-based power system protection are difficult to compare because task definitions, measurement access, data partitions, metrics, and generalization conditions often differ. EvEMTBench addresses this gap with an open, executable, and versioned benchmark that fixes these evaluation choices while leaving model design open. Across four grids spanning 20-345 kV, it defines 12 protection and event-analysis functions instantiated as 24 scored tasks and supports structured evaluation across observability conditions, predefined distribution shifts, and zero-shot and fine-tuned cross-grid transfer. Committed partitions, leakage controls, and reproducible reporting provide a common basis for comparing future methods. A reference evaluation spanning trivial, conventional, feature-based, and deep-learning baselines shows that wider observability is not uniformly beneficial, shifted conditions can reveal failures not apparent in-distribution, and cross-grid transfer is substantially stronger for fault detection than for fault localization. Protection-relevant diagnostics identify failure modes not apparent from primary metrics alone. EvEMTBench therefore makes generalization in machine-learning-based protection an explicit and reproducible evaluation problem.