# Conducting Reliable Keyword Estimation with Unreliable ASR in Police Audio Data

arXiv: https://arxiv.org/abs/2610.26614 · HTML: https://mlsanity.com/paper/2610.26614/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in ASR evaluation, bias and atypical speech, one of 11 directions in Speech and audio language models.
- Builds on 11 of its 19 references that are in the map.
- First posted on arXiv 19 Oct 2026.

## Placed in

- Speech and audio language models / [ASR evaluation, bias and atypical speech](https://mlsanity.com/speech/asr-evaluation-bias-and-atypical-speech/)

## Builds on (most cited first, 11 of 11)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Prediction-Powered Inference](https://mlsanity.com/paper/2301.09633/) (2023)
- [Qwen3-ASR Technical Report](https://mlsanity.com/paper/2601.21337/) (2026)
- [PPI++: Efficient Prediction-Powered Inference](https://mlsanity.com/paper/2311.01453/) (2023)
- [Can Unconfident LLM Annotations Be Used for Confident Conclusions?](https://mlsanity.com/paper/2408.15204/) (2024)
- [Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models](https://mlsanity.com/paper/2306.04746/) (2023)
- [A Note on the Prediction-Powered Bootstrap](https://mlsanity.com/paper/2405.18379/) (2024)
- [Extracting Biomedical Entities from Noisy Audio Transcripts](https://mlsanity.com/paper/2403.17363/) (2024)
- [The Power of Prosody and Prosody of Power: An Acoustic Analysis of Finnish Parliamentary Speech](https://mlsanity.com/paper/2305.16040/) (2023)
- [Speech Recognition for Analysis of Police Radio Communication](https://mlsanity.com/paper/2409.10858/) (2024)
- [Pretrained ASR Pseudo-labeling for Noisy Police Audio](https://mlsanity.com/paper/2609.30469/) (2026)

## Abstract (arXiv)

Extracting insights from large audio data sets using human transcription is expensive and time-consuming. Off-the-shelf Automatic Speech Recognition (ASR) systems offer an easy-to-use scalable alternative, but systematic transcription errors, especially in domain-specific settings, can bias downstream analyses and lead to unreliable conclusions. We investigate whether imperfect ASR transcripts can still be used to obtain statistically valid keyword frequency estimates by combining model outputs with a small set of human-labeled examples, following Prediction Powered Inference and Design-based Supervised Learning frameworks. We evaluate this approach on police radio communications from Baltimore and Chicago, a domain in which keyword distributions could provide valuable insights into police activity, but off-the-shelf ASR systems exhibit systematic errors on operationally important terminology. Using bootstrap-based confidence interval coverage as our evaluation metric, we find that our approach substantially increases the number of domain-specific keywords that can be analyzed with statistically valid confidence intervals, from 43.0% using ASR alone and 27.2% using only a small set of human-labeled transcriptions to 56.2% in the Baltimore dataset, with similar improvements observed in Chicago data. Our results demonstrate that post-hoc error correction can improve statistical validity of downstream analyses even for poor-quality transcriptions, offering an avenue for better leveraging off-the-shelf ASR models for large-scale speech analysis.