# Finding Label and Model Errors in Perception Data With Learned Observation Assertions

arXiv: https://arxiv.org/abs/2201.05797 · HTML: https://mlsanity.com/paper/2201.05797/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jan 2022.

## Built on by (most cited first, 6 of 6)

- [Joint Neural Architecture and Hyperparameter Search for Correlated Time Series Forecasting](https://mlsanity.com/paper/2211.16126/) (2022)
- [A Comprehensive Survey of Deep Transfer Learning for Anomaly Detection in Industrial Time Series: Methods, Applications, and Directions](https://mlsanity.com/paper/2307.05638/) (2023)
- [SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines](https://mlsanity.com/paper/2401.03038/) (2024)
- [Identifying Label Errors in Object Detection Datasets by Loss Inspection](https://mlsanity.com/paper/2303.06999/) (2023)
- [Deep Learning Safety Concerns in Automated Driving Perception](https://mlsanity.com/paper/2309.03774/) (2023)
- [Detecting Errors in a Numerical Response via any Regression Model](https://mlsanity.com/paper/2305.16583/) (2023)

## Abstract (arXiv)

ML is being deployed in complex, real-world scenarios where errors have impactful consequences. In these systems, thorough testing of the ML pipelines is critical. A key component in ML deployment pipelines is the curation of labeled training data. Common practice in the ML literature assumes that labels are the ground truth. However, in our experience in a large autonomous vehicle development center, we have found that vendors can often provide erroneous labels, which can lead to downstream safety risks in trained models. To address these issues, we propose a new abstraction, learned observation assertions, and implement it in a system called Fixy. Fixy leverages existing organizational resources, such as existing (possibly noisy) labeled datasets or previously trained ML models, to learn a probabilistic model for finding errors in human- or model-generated labels. Given user-provided features and these existing resources, Fixy learns feature distributions that specify likely and unlikely values (e.g., that a speed of 30mph is likely but 300mph is unlikely). It then uses these feature distributions to score labels for potential errors. We show that FIxy can automatically rank potential errors in real datasets with up to 2× higher precision compared to recent work on model assertions and standard techniques such as uncertainty sampling.