# Measurement Under Selection: Decoy-Calibrated Failure Audits for Language Models

arXiv: https://arxiv.org/abs/2606.09046 · HTML: https://mlsanity.com/paper/2606.09046/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 8 Jun 2026.

## Builds on (most cited first, 15 of 15)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://mlsanity.com/paper/2412.15204/) (2024)
- [Domino: Discovering Systematic Errors with Cross-Modal Embeddings](https://mlsanity.com/paper/2203.14960/) (2022)
- [Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection](https://mlsanity.com/paper/1610.02351/) (2016)
- [Controlling the false discovery rate via knockoffs](https://mlsanity.com/paper/1404.5609/) (2014)
- [The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Models](https://mlsanity.com/paper/2107.00758/) (2021)
- [Statistical Inference for Fairness Auditing](https://mlsanity.com/paper/2305.03712/) (2023)
- [HiBug2: Efficient and Interpretable Error Slice Discovery for Comprehensive Model Debugging](https://mlsanity.com/paper/2501.16751/) (2025)
- [Error Discovery by Clustering Influence Embeddings](https://mlsanity.com/paper/2312.04712/) (2023)
- [LADDER: Language-Driven Slice Discovery and Error Rectification in Vision Classifiers](https://mlsanity.com/paper/2408.07832/) (2024)
- [Error Slice Discovery via Manifold Compactness](https://mlsanity.com/paper/2501.19032/) (2025)
- [The 99% Success Paradox: When Near-Perfect Retrieval Equals Random Selection](https://mlsanity.com/paper/2605.18857/) (2026)
- [FairTree: Subgroup Fairness Auditing of Machine Learning Models with Bias-Variance Decomposition](https://mlsanity.com/paper/2604.19357/) (2026)

## Built on by (most cited first, 1 of 1)

- [Selection-Aware Stress Testing for Interactive Agents](https://mlsanity.com/paper/2608.30916/) (2026)

## Abstract (arXiv)

Knowing how often a language model fails does not explain where its errors concentrate. When auditors examine many explanations, the strongest observed pattern may arise by chance. We introduce Janus, a procedure for checking proposed error patterns before reporting them. Janus starts with a fixed list of yes/no properties of the examples being evaluated, such as whether the input is long. For each property, it compares the model's error rates on examples with that property and those without it. To see how large a difference can arise by chance, it repeats this calculation after shuffling the yes/no labels across examples without changing the group sizes. These shuffled properties are called decoys. A pattern is reported only if the size of its error difference meets a threshold set using decoys. On separate held-out examples, the same group must still have the higher error rate and the difference must meet a minimum, which was chosen in advance. In a controlled experiment, where the model must find a code in documents containing tables of staff, projects, and renewal codes, Janus confirms five related patterns of higher error rates on tasks requiring more lookups across tables. It also confirms a sixth pattern: lower error rates on examples with the needed information at the ends of the tables. In our samples from the MuSiQue and LongBench v2 public benchmarks, SliceLine finds groups with high error rates, while Janus reports no confirmed error patterns for the example properties we chose to test. For comparison, we use standard tests that shuffle errors and account for testing many candidates. With the same holdout check, they confirm two to six controlled patterns, depending on the test and threshold, and none on either benchmark. In simulations with no real error patterns, Janus reports false patterns more often than Benjamini-Hochberg, depending on the decoy count.