# When Do LLMs Actually Help? Evaluating LLMs as Data Quality Annotators

arXiv: https://arxiv.org/abs/2608.18158 · HTML: https://mlsanity.com/paper/2608.18158/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 13 references that are in the map.
- First posted on arXiv 13 Aug 2026.

## Builds on (most cited first, 12 of 12)

- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Match, Compare, or Select? An Investigation of Large Language Models for Entity Matching](https://mlsanity.com/paper/2405.16884/) (2024)
- [(Almost) All of Entity Resolution](https://mlsanity.com/paper/2008.04443/) (2020)
- [Large Language Models as Financial Data Annotators: A Study on Effectiveness and Efficiency](https://mlsanity.com/paper/2403.18152/) (2024)
- [Do Repetitions Matter? Strengthening Reliability in LLM Evaluations](https://mlsanity.com/paper/2509.24086/) (2025)
- [Multi-stage Large Language Model Pipelines Can Outperform GPT-4o in Relevance Assessment](https://mlsanity.com/paper/2501.14296/) (2025)
- [Automatic Validation of Textual Attribute Values in E-commerce Catalog by Learning with Limited Labeled Data](https://mlsanity.com/paper/2006.08779/) (2020)
- [Auto prompting without training labels: An LLM cascade for product quality assessment in e-commerce catalogs](https://mlsanity.com/paper/2510.23941/) (2025)
- [Evaluating Prompting Strategies and Large Language Models in Systematic Literature Review Screening: Relevance and Task-Stage Classification](https://mlsanity.com/paper/2510.16091/) (2025)
- [The Necessity of Setting Temperature in LLM-as-a-Judge](https://mlsanity.com/paper/2603.28304/) (2026)
- [Structured Multi-Step Reasoning for Entity Matching Using Large Language Model](https://mlsanity.com/paper/2511.22832/) (2025)
- [Towards Consistent Detection of Cognitive Distortions: LLM-Based Annotation and Dataset-Agnostic Evaluation](https://mlsanity.com/paper/2511.01482/) (2025)

## Abstract (arXiv)

LLMs have been increasingly used to catch data quality issues automatically, but we know very little about how consistent these judgments actually are. This study tests an LLM on two e-commerce data quality tasks, entity matching and brand mislabeling, against rule based baselines and human verified ground truth, under both zero-shot and few-shot prompting. On entity matching while using the Abt Buy benchmark (2,194 labeled pairs), a simple rule based baseline (F1=0.950) performed about as well as LLM zero shot prompting (F1=0.948). Moreover, a few-shot prompt revision that looked effective on a small validation sample reduced full-scale performance to F1=0.914. This showed that small sample prompt evaluation can be misleading. On brand mislabeling detection, using 500 Amazon product listings with synthetically injected labeling errors, the LLM clearly outperformed a naive rule based baseline (F1=0.833 vs 0.721), because it could draw on background knowledge of brand product relationships that a simple rule could not access. Testing consistency across repeated runs (200 pairs, 5 runs at temperature 0.7) showed the model agreeing with itself 99.7% of the time on average, with 99% of pairs giving identical answers across all 5 runs. Using majority voting across these runs only improved F1 by 0.005, at 5 times the inference cost. These results suggest that the value of using an LLM over traditional methods depends heavily on the task. LLMs offer little advantage when strong lexical signals already exist, but a clear advantage when the task requires background knowledge, all while remaining highly consistent across repeated queries.