# The Selection Rule Decides the Winner: A Pre-Registered Audit of Open-Set Graph Anomaly Detection

arXiv: https://arxiv.org/abs/2609.33370 · HTML: https://mlsanity.com/paper/2609.33370/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 62 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Builds on (most cited first, 29 of 29)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Pitfalls of Graph Neural Network Evaluation](https://mlsanity.com/paper/1811.05868/) (2018)
- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](https://mlsanity.com/paper/2110.08263/) (2021)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [A critical look at the evaluation of GNNs under heterophily: Are we really making progress?](https://mlsanity.com/paper/2302.11640/) (2023)
- [Revisiting Graph Neural Networks: All We Have is Low-Pass Filters](https://mlsanity.com/paper/1905.09550/) (2019)
- [Enhancing Graph Neural Network-based Fraud Detectors against Camouflaged Fraudsters](https://mlsanity.com/paper/2008.08692/) (2020)
- [A Fair Comparison of Graph Neural Networks for Graph Classification](https://mlsanity.com/paper/1912.09893/) (2019)
- [A Comprehensive Survey on Graph Anomaly Detection with Deep Learning](https://mlsanity.com/paper/2106.07178/) (2021)
- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [Rethinking Graph Neural Networks for Anomaly Detection](https://mlsanity.com/paper/2205.15508/) (2022)
- [Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning](https://mlsanity.com/paper/2103.00113/) (2021)
- [Show Your Work: Improved Reporting of Experimental Results](https://mlsanity.com/paper/1909.03004/) (2019)
- [From Amateurs to Connoisseurs: Modeling the Evolution of User Expertise through Online Reviews](https://mlsanity.com/paper/1303.4402/) (2013)
- [Catching Both Gray and Black Swans: Open-set Supervised Anomaly Detection](https://mlsanity.com/paper/2203.14506/) (2022)
- [AnomalyDAE: Dual autoencoder for anomaly detection on attributed networks](https://mlsanity.com/paper/2002.03665/) (2020)
- [BOND: Benchmarking Unsupervised Outlier Node Detection on Static Attributed Graphs](https://mlsanity.com/paper/2206.10071/) (2022)
- [GADBench: Revisiting and Benchmarking Supervised Graph Anomaly Detection](https://mlsanity.com/paper/2306.12251/) (2023)
- [Truncated Affinity Maximization: One-class Homophily Modeling for Graph Anomaly Detection](https://mlsanity.com/paper/2306.00006/) (2023)
- [Deep Graph Anomaly Detection: A Survey and New Perspectives](https://mlsanity.com/paper/2409.09957/) (2024)
- [Generative Semi-supervised Graph Anomaly Detection](https://mlsanity.com/paper/2402.11887/) (2024)
- [SpaceGNN: Multi-Space Graph Neural Network for Node Anomaly Detection with Extremely Limited Labels](https://mlsanity.com/paper/2502.03201/) (2025)
- [Open-Set Graph Anomaly Detection via Normal Structure Regularisation](https://mlsanity.com/paper/2311.06835/) (2023)

## Abstract (arXiv)

Open-set graph anomaly detection trains on a few labeled anomalies from one class and must also find anomaly classes that were never labeled. Published results share three conventions: the test score is read at the best epoch on the test set, baseline numbers are copied from earlier papers, and most anomalies are minority classes relabeled as anomalous. We ask how much of the reported ranking these conventions decide. We re-run two recent methods, DEMO and NSReg, together with OUTPOST, a small first-order detector built for this study. All three use one protocol with identical seeds and splits on eight graphs (seven for the baselines, which cannot run on ogbn-mag), ten seeds each, and every run is scored under both the best-epoch rule and a deployable validation rule. Before the runs that test them, we registered 40 predictions. Three findings hold. First, the rule changes the leader: under the best-epoch rule, OUTPOST and NSReg each lead three of seven graphs, while under the validation rule, NSReg leads five. Second, the best-epoch bonus depends on how the benchmark was built: 0.045--0.080 AUC-ROC on the three small relabeled-class graphs and 0.002--0.014 on the three real fraud graphs. Third, pseudo-labeling in OUTPOST is worth 0.038--0.065 AUC-ROC on the same three graphs but gives no benefit on any real fraud graph. We also show that a 0.002 tie band for hyperparameter selection lies below the paired standard error on all six graphs tested, even at ten seeds. Twelve of our 40 predictions were falsified, and we report them. We close with a short reporting checklist.