Curating Merchant-Matching Training Data with Two Confidence-Gated Local LLM Judges
Not on the map. Why?
Abstract (arXiv)
Merchant matching resolves a noisy payment descriptor to a retrieved merchant entity or returns no match. A key challenge in curating training labels is distinguishing teacher abstention from evidence that no acceptable entity exists: false no-match labels contaminate pseudo-labeled data, while conservative labeling reduces coverage. We investigate whether agreement between two local large language model judges improves pseudo-label reliability. A label is retained only when the judges agree, with separate ordered thresholds for selections and abstentions that guarantee disjoint positive and negative label sets. Retrospective replay on 2,000 expert-annotated queries shows that higher selection thresholds can improve positive-label purity, whereas higher abstention thresholds increase false no-match labels. At thresholds (0.86, 0.80), Muse Glimmer 30B and Gemma 4 31B jointly label 1,633 queries (81.7% coverage) at 96.88% purity; positive and negative purities are 99.47% and 93.38%. This exceeds either constituent model at the same thresholds by more than two percentage points, with lower coverage. A split-half check finds only 0.14 percentage points of threshold-selection optimism. A symmetric threshold of 0.86 adds 40 erroneous no-match labels, while 46 false abstentions persist even with no confidence threshold. Across five matched within-model comparisons, higher reasoning effort yields no clear F0.5 gain and increases median latency by 1.8-5.0 times. These results motivate separate thresholding and auditing for positive and negative pseudo-labels. The study establishes label purity, not student utility; fresh-data curation and student fine-tuning remain necessary to demonstrate downstream value.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 22 references that are in the map. First posted on arXiv 27 Sep 2026.
Builds on
14 of 22 references.
References · 14
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
- 2017Selective Classification for Deep Neural Networkscited by 852
- 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 372
- 2026Gemma 4 Technical Reportcited by 259
- 2024Match, Compare, or Select? An Investigation of Large Language Models for Entity Matchingcited by 32
- 2023Entity Matching using Large Language Modelscited by 31
- 2024Fine-tuning Large Language Models for Entity Matchingcited by 16
- 2023Cost-Effective In-Context Learning for Entity Resolution: A Design Space Explorationcited by 15
- 2026Task Complexity Matters: An Empirical Study of Reasoning in LLMs for Sentiment Analysiscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
- 2026Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency
- 2026MERGED: Multimodal Entity Resolution via Generated Expert Reasoning Distillation
- 2025Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs
- 2026Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models
- 2026Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdict
- 2026SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decomposition
- 2024Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data