EvEMTBench: An Open Benchmark for Machine Learning in Power System Protection

Studies of machine-learning-based power system protection are difficult to compare due to varying task definitions, measurement access, data partitions, metrics, and generalization conditions. This paper presents EvEMTBench, an open, executable, versioned benchmark designed to standardize these evaluation choices while maintaining flexibility in model design. The benchmark covers four grids with voltage levels from 20 to 345 kV, defining 12 protection and event-analysis functions as 24 scored tasks. It supports structured evaluation across a range of observability conditions, predefined distribution shifts, and zero-shot and fine-tuned cross-grid transfer. With committed partitions, leakage controls, and reproducible reporting, EvEMTBench establishes a common framework for comparing future methodologies. A reference evaluation featuring trivial, conventional, feature-based, and deep-learning baselines reveals that wider observability does not always yield benefits, shifted conditions can expose failures not visible in original distributions, and cross-grid transfer is significantly more effective for fault detection than for fault localization. Additionally, protection-relevant diagnostics uncover failure modes not evident from primary metrics alone. Consequently, EvEMTBench explicitly frames generalization in machine-learning-based protection as a reproducible evaluation challenge.