# Assessing the Impact of the Quality of Textual Data on Feature Representation and Machine Learning Models

arXiv: https://arxiv.org/abs/2502.08669 · HTML: https://mlsanity.com/paper/2502.08669/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 74 references that are in the map.
- First posted on arXiv 12 Feb 2025.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Improving Text Embeddings with Large Language Models](https://mlsanity.com/paper/2401.00368/) (2023)
- [GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records](https://mlsanity.com/paper/2203.03540/) (2022)
- [MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval](https://mlsanity.com/paper/2307.00589/) (2023)
- [A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction](https://mlsanity.com/paper/2208.10240/) (2022)
- [Grammatical Error Correction: A Survey of the State of the Art](https://mlsanity.com/paper/2211.05166/) (2022)
- [Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss](https://mlsanity.com/paper/1902.09191/) (2019)
- [Outlier Detection for Improved Data Quality and Diversity in Dialog Systems](https://mlsanity.com/paper/1904.03122/) (2019)
- [Natural Language Processing for EHR-Based Computational Phenotyping](https://mlsanity.com/paper/1806.04820/) (2018)
- [Data Quality Toolkit: Automatic assessment of data quality and remediation for machine learning datasets](https://mlsanity.com/paper/2108.05935/) (2021)
- [Data Readiness Report](https://mlsanity.com/paper/2010.07213/) (2020)
- [Evolutionary Data Measures: Understanding the Difficulty of Text Classification Tasks](https://mlsanity.com/paper/1811.01910/) (2018)
- [The Battle of LLMs: A Comparative Study in Conversational QA Tasks](https://mlsanity.com/paper/2405.18344/) (2024)
- [Beyond ChatGPT: Enhancing Software Quality Assurance Tasks with Diverse LLMs and Validation Techniques](https://mlsanity.com/paper/2409.01001/) (2024)

## Abstract (arXiv)

Background: Data collected in controlled settings typically results in high-quality datasets. However, in real-world applications, the quality of data collection is often compromised. It is well established that the quality of a dataset significantly impacts the performance of machine learning models. Methods: A rudimentary error rate metric was developed to evaluate textual dataset quality at the token level. Mixtral Large Language Model (LLM) was used to quantify and correct errors in low quality datasets. The study analyzed two healthcare datasets: the high-quality MIMIC-III public hospital dataset and a lower-quality private dataset from Australian aged care homes. Errors were systematically introduced into MIMIC at varying rates, while the ACH dataset quality was improved using the LLM. Results: For the sampled 35,774 and 6,336 patients from the MIMIC and ACH datasets respectively, we used Mixtral to introduce errors in MIMIC and correct errors in ACH. Mixtral correctly detected errors in 63% of progress notes, with 17% containing a single token misclassified due to medical terminology. LLMs demonstrated potential for improving progress note quality by addressing various errors. Under varying error rates, feature representation performance was tolerant to lower error rates (<10%) but declined significantly at higher rates. Conclusions: The study revealed that models performed relatively well on datasets with lower error rates (<10%), but their performance declined significantly as error rates increased (>=10%). Therefore, it is crucial to evaluate the quality of a dataset before utilizing it for machine learning tasks. For datasets with higher error rates, implementing corrective measures is essential to ensure the reliability and effectiveness of machine learning models.