# The record is part of the task: matched-record evaluation of text classifiers across maintenance, safety and recall reporting

arXiv: https://arxiv.org/abs/2609.16267 · HTML: https://mlsanity.com/paper/2609.16267/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 46 references that are in the map.
- First posted on arXiv 14 Sep 2026.

## Builds on (most cited first, 12 of 12)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [Characterizing the Value of Information in Medical Notes](https://mlsanity.com/paper/2010.03574/) (2020)
- [MaintNet: A Collaborative Open-Source Library for Predictive Maintenance Language Resources](https://mlsanity.com/paper/2005.12443/) (2020)
- [Backdoor Adjustment of Confounding by Provenance for Robust Text Classification of Multi-institutional Clinical Notes](https://mlsanity.com/paper/2310.02451/) (2023)
- [Data Issues in Industrial AI System: A Meta-Review and Research Strategy](https://mlsanity.com/paper/2406.15784/) (2024)
- [Applications of natural language processing in aviation safety: A review and qualitative analysis](https://mlsanity.com/paper/2501.06210/) (2025)
- [Phase of Flight Classification in Aviation Safety using LSTM, GRU, and BiLSTM: A Case Study with ASN Dataset](https://mlsanity.com/paper/2501.07925/) (2025)
- [Hybrid Agentic AI and Multi-Agent Systems in Smart Manufacturing](https://mlsanity.com/paper/2511.18258/) (2025)
- [Document Classification Pattern Recognition via Information Fusion: A Systematic Review of Multimodal and Multiview Representation Approaches](https://mlsanity.com/paper/2605.23910/) (2026)

## Abstract (arXiv)

Many operational cases are documented more than once, at different workflow stages and for different purposes, yet model evaluations normally select one of these records before model comparison begins. We treat that selection as part of the evaluation and compare matched records of the same cases under fixed labels and splits in three systems: GE Aerospace repair events, NASA ASRS safety reports and NHTSA vehicle recalls. The design fixes, for any decision a classifier would support, the records that existed and could be read when it was taken, and compares them retrospectively, as written. Across the three GE fields, for events whose label comes from parts transactions independently of the narratives, held-out macro-F1 ranged from 0.33 to 0.91. A difference of 0.46 separated the customer report, written before shop work, from the technician report, written after diagnosis but before the transaction that generates the label. That difference is substantially larger than the representation and architecture differences tested on the same events. The public systems showed different patterns: the NHTSA defect summary remained strongest under every model family tested, whereas the ASRS analyst synopsis outperformed the reporter narrative under learned sequence models but not under lexical baselines. Secondary analyses showed that some model comparisons were also record-dependent. Evaluations should be run on the information available at the intended decision point and should report how both the record and the label were produced.