# Do LLMs Judge Distantly Supervised Named Entity Labels Well? Constructing the JudgeWEL Dataset

arXiv: https://arxiv.org/abs/2601.00411 · HTML: https://mlsanity.com/paper/2601.00411/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 62 references that are in the map.
- First posted on arXiv 1 Jan 2026.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [Empirical Study of Zero-Shot NER with ChatGPT](https://mlsanity.com/paper/2310.10035/) (2023)
- [POLYGLOT-NER: Massive Multilingual Named Entity Recognition](https://mlsanity.com/paper/1410.3791/) (2014)
- [Prompt-Based Metric Learning for Few-Shot NER](https://mlsanity.com/paper/2211.04337/) (2022)
- [LinkNER: Linking Local Named Entity Recognition Models to Large Language Models using Uncertainty](https://mlsanity.com/paper/2402.10573/) (2024)
- [LuxBank: The First Universal Dependency Treebank for Luxembourgish](https://mlsanity.com/paper/2411.04813/) (2024)
- [Neural Text Normalization for Luxembourgish using Real-Life Variation Data](https://mlsanity.com/paper/2412.09383/) (2024)
- [Overview of the First Workshop on Language Models for Low-Resource Languages (LoResLM 2025)](https://mlsanity.com/paper/2412.16365/) (2024)

## Built on by (most cited first, 1 of 1)

- [ltzGLUE: Luxembourgish General Language Understanding Evaluation](https://mlsanity.com/paper/2604.17976/) (2026)

## Abstract (arXiv)

We present judgeWEL, a dataset for named entity recognition (NER) in Luxembourgish, automatically labelled and subsequently verified using large language models (LLM) in a novel pipeline. Building datasets for under-represented languages remains one of the major bottlenecks in natural language processing, where the scarcity of resources and linguistic particularities make large-scale annotation costly and potentially inconsistent. To address these challenges, we propose and evaluate a novel approach that leverages Wikipedia and Wikidata as structured sources of weak supervision. By exploiting internal links within Wikipedia articles, we infer entity types based on their corresponding Wikidata entries, thereby generating initial annotations with minimal human intervention. Because such links are not uniformly reliable, we mitigate noise by employing and comparing several LLMs to identify and retain only high-quality labelled sentences. The resulting corpus is approximately five times larger than the currently available Luxembourgish NER dataset and offers broader and more balanced coverage across entity categories, providing a substantial new resource for multilingual and low-resource NER research.