# Recovery Rates Are Not Comparable Across Transcription Factors: Chance Correction for Attribution Evaluation

arXiv: https://arxiv.org/abs/2609.16271 · HTML: https://mlsanity.com/paper/2609.16271/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 24 references that are in the map.
- First posted on arXiv 31 Jul 2026.

## Builds on (most cited first, 6 of 6)

- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution](https://mlsanity.com/paper/2306.15794/) (2023)
- [DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA](https://mlsanity.com/paper/2412.05430/) (2024)
- [Monte Carlo Null Models for Genomic Data](https://mlsanity.com/paper/1404.5970/) (2014)
- [Position: Genomic Model Research Must Move Beyond Anecdotal Evaluation of Interpretability Methods](https://mlsanity.com/paper/2606.07607/) (2026)
- [Evaluating Post-hoc Explanations of the Transformer-based Genome Language Model DNABERT-2](https://mlsanity.com/paper/2604.21690/) (2026)

## Abstract (arXiv)

Attribution methods for genomic sequence models are commonly evaluated by how much of a known motif they recover, or by how a prediction degrades as evidence is deleted. Neither score is interpretable without the value it would take by chance, and neither is routinely reported against one. We show that this omission is not a matter of precision but of validity. The uniform chance level for contiguous motif overlap is (L/(N-L+1)); across 268 transcription factors in UniBind it ranges from 0.0118 to 0.0427, a 3.6-fold spread determined by motif length and window size alone. For two factors the bootstrap intervals of the chance levels themselves do not overlap, so their raw recovery rates are not comparable quantities. Correcting for this dissolves a published three-way classification of five factors: a factor reported as a resolution failure attains the second-highest corrected value, ahead of one of the two positive controls, and two reported as complete failures fall at or below chance. We further show that perturbation-based evaluation can fail its own precondition: for one factor a fully masked input still scores above the decision boundary, and the curve is not monotone in the number of masked positions, so the area under it is not a measure of faithfulness. We provide chance levels in closed form, a chance-corrected score, and two screens that run before any attribution is computed.