# deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networks

arXiv: https://arxiv.org/abs/2204.06815 · HTML: https://mlsanity.com/paper/2204.06815/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 54 references that are in the map.
- First posted on arXiv 14 Apr 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [Deep Reinforcement Learning and the Deadly Triad](https://mlsanity.com/paper/1812.02648/) (2018)
- [Show Your Work: Improved Reporting of Experimental Results](https://mlsanity.com/paper/1909.03004/) (2019)
- [Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning](https://mlsanity.com/paper/1811.12808/) (2018)
- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [With Little Power Comes Great Responsibility](https://mlsanity.com/paper/2010.06595/) (2020)
- [The Benchmark Lottery](https://mlsanity.com/paper/2107.07002/) (2021)
- [Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers](https://mlsanity.com/paper/2007.01547/) (2020)
- [Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis](https://mlsanity.com/paper/1606.04316/) (2016)
- [Do Transformer Modifications Transfer Across Implementations and Applications?](https://mlsanity.com/paper/2102.11972/) (2021)
- [Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers](https://mlsanity.com/paper/2106.15195/) (2021)
- [An optimal transportation approach for assessing almost stochastic order](https://mlsanity.com/paper/1705.01788/) (2017)
- [Hyperparameter Optimization Is Deceiving Us, and How to Stop It](https://mlsanity.com/paper/2102.03034/) (2021)
- [Why Comparing Single Performance Scores Does Not Allow to Draw Conclusions About Machine Learning Approaches](https://mlsanity.com/paper/1803.09578/) (2018)
- [Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses](https://mlsanity.com/paper/1911.03850/) (2019)
- [Models for the assessment of treatment improvement: the ideal and the feasible](https://mlsanity.com/paper/1612.01291/) (2016)

## Built on by (most cited first, 28 of 28)

- [LingMess: Linguistically Informed Multi Expert Scorers for Coreference Resolution](https://mlsanity.com/paper/2205.12644/) (2022)
- [Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity](https://mlsanity.com/paper/2210.15452/) (2022)
- [A noisy elephant in the room: Is your out-of-distribution detector robust to label noise?](https://mlsanity.com/paper/2404.01775/) (2024)
- [On Retrospective k-space Subsampling schemes For Deep MRI Reconstruction](https://mlsanity.com/paper/2301.08365/) (2023)
- [VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning](https://mlsanity.com/paper/2309.15494/) (2023)
- [DynDepNet: Learning Time-Varying Dependency Structures from fMRI Data via Dynamic Graph Structure Learning](https://mlsanity.com/paper/2209.13513/) (2022)
- [Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech](https://mlsanity.com/paper/2305.16406/) (2023)
- [Tokenization with Factorized Subword Encoding](https://mlsanity.com/paper/2306.07764/) (2023)
- [Class Uncertainty: A Measure to Mitigate Class Imbalance](https://mlsanity.com/paper/2311.14090/) (2023)
- [Generalized Attention Flow: Feature Attribution for Transformer Models via Maximum Flow](https://mlsanity.com/paper/2502.15765/) (2025)
- [Risk Aware Benchmarking of Large Language Models](https://mlsanity.com/paper/2310.07132/) (2023)
- [Collaboration or Corporate Capture? Quantifying NLP's Reliance on Industry Artifacts and Contributions](https://mlsanity.com/paper/2312.03912/) (2023)
- [End-to-end Adaptive Dynamic Subsampling and Reconstruction for Cardiac MRI](https://mlsanity.com/paper/2403.10346/) (2024)
- [Multivariate Stochastic Dominance via Optimal Transport and Applications to Models Benchmarking](https://mlsanity.com/paper/2406.06425/) (2024)
- [pAI/MSc: ML Theory Research with Humans on the Loop](https://mlsanity.com/paper/2604.20622/) (2026)
- [Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers](https://mlsanity.com/paper/2606.02776/) (2026)
- [DBGDGM: Dynamic Brain Graph Deep Generative Model](https://mlsanity.com/paper/2301.11408/) (2023)
- [Triggering Dark Showers with Conditional Dual Auto-Encoders](https://mlsanity.com/paper/2306.12955/) (2023)
- [NeuralSolver: Learning Algorithms For Consistent and Efficient Extrapolation Across General Tasks](https://mlsanity.com/paper/2402.15393/) (2024)
- [A multimodal LLM for the non-invasive decoding of spoken text from brain recordings](https://mlsanity.com/paper/2409.19710/) (2024)
- [Tail-Shape Estimation in LLM Evaluation Is Fragile: A Protocol for Diagnosing False Positives](https://mlsanity.com/paper/2606.16511/) (2026)
- [Estimating Motor Symptom Presence and Severity in Parkinson's Disease from Wrist Accelerometer Time Series using ROCKET and InceptionTime](https://mlsanity.com/paper/2304.11265/) (2023)
- [A Target-Aware Analysis of Data Augmentation for Hate Speech Detection](https://mlsanity.com/paper/2410.08053/) (2024)
- [Enriching Social Science Research via Survey Item Linking](https://mlsanity.com/paper/2412.15831/) (2024)
- [Modelling Adjectival Modification Effects on Semantic Plausibility](https://mlsanity.com/paper/2507.21828/) (2025)
- [CorefInst: Leveraging LLMs for Multilingual Coreference Resolution](https://mlsanity.com/paper/2509.17505/) (2025)
- [ManufactuBERT: Efficient Continual Pretraining for Manufacturing](https://mlsanity.com/paper/2511.05135/) (2025)
- [DEEP: Docker-based Execution and Evaluation Platform](https://mlsanity.com/paper/2602.19583/) (2026)

## Abstract (arXiv)

A lot of Machine Learning (ML) and Deep Learning (DL) research is of an empirical nature. Nevertheless, statistical significance testing (SST) is still not widely used. This endangers true progress, as seeming improvements over a baseline might be statistical flukes, leading follow-up research astray while wasting human and computational resources. Here, we provide an easy-to-use package containing different significance tests and utility functions specifically tailored towards research needs and usability.