# MASH-Bench: Diagnosing Cross-Source Failure in Mass-Shooting Risk Classification

arXiv: https://arxiv.org/abs/2608.22460 · HTML: https://mlsanity.com/paper/2608.22460/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 20 references that are in the map.
- First posted on arXiv 23 Aug 2026.

## Builds on (most cited first, 8 of 8)

- [Return of Frustratingly Easy Domain Adaptation](https://mlsanity.com/paper/1511.05547/) (2015)
- [Confounding variables can degrade generalization performance of radiological deep learning models](https://mlsanity.com/paper/1807.00431/) (2018)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [FairGAN: Fairness-aware Generative Adversarial Networks](https://mlsanity.com/paper/1805.11202/) (2018)
- [Benchmarking Distribution Shift in Tabular Data with TableShift](https://mlsanity.com/paper/2312.07577/) (2023)
- [Pedestrian Detection: Domain Generalization, CNNs, Transformers and Beyond](https://mlsanity.com/paper/2201.03176/) (2022)
- [On the Cross-Dataset Generalization of Machine Learning for Network Intrusion Detection](https://mlsanity.com/paper/2402.10974/) (2024)
- [On Big Data Benchmarking](https://mlsanity.com/paper/1402.5194/) (2014)

## Abstract (arXiv)

Public mass-shooting databases differ substantially in coverage, feature availability, and reporting practices, creating challenges for machine-learning models that must generalize across data sources. We introduce MASH-Bench, a harmonized benchmark of 6,968 incidents from four U.S. databases: Kaggle, Mother Jones, Stanford MSA, and the Gun Violence Archive (GVA). We evaluate cross-source risk classification using leave-one-dataset-out (LODO) evaluation. Random Forest, XGBoost, and LightGBM achieve VeryHigh-risk recall of 0.68-0.89 on the curated sources but generalize poorly to GVA, where mean recall drops to 0.20 and precision to 0.0004. To investigate the source of this degradation, we conduct a controlled feature-masking ablation that removes the five features unavailable in GVA from the curated sources. The resulting recall collapse to zero provides evidence that feature completeness is a major contributor to the observed cross-source failure. We further evaluate three domain-adaptation approaches: DANN, CORAL, and importance weighting. DANN improves VeryHigh-risk recall on GVA by 0.282 (95% CI [0.11, 0.47], p = 0.003), although precision remains low, whereas CORAL and importance weighting yield zero recall. Oracle prior-shift recalibration likewise fails to recover VeryHigh-risk predictions, indicating that label-side correction alone is insufficient under the observed feature deficiencies. A per-group audit further identifies substantial disparities associated with media-attributed mental-health labels. Overall, these results indicate that, in MASH-Bench, cross-source generalization is constrained more by feature completeness and label prevalence than by classifier choice. The benchmark provides a controlled setting for diagnosing these effects in cross-source risk classification.