No PUN Intended: Plausible Unknown Names for Person-Centred LLM Evaluation
Not on the map. Why?
Abstract (arXiv)
Person names are widely used as prompt variables in LLM evaluations of factuality, privacy leakage, bias and abstention, but when a name's evidential status is uncontrolled, measurements may conflate memorisation, retrieval, name priors and wrong-person attribution. We operationalise an unknown name as one with plausible First-Last form, no indexed full-name evidence, and no ambiguity signals under a documented validation run, and introduce PUN (Plausible Unknown Names), a protocol for constructing and validating such names, combining Wikidata-derived components, web-enabled LLM screening, and controlled search revalidation. We report acceptance rate, reproducibility, ablations, and a 204-participant human study, finding accepted names are more name-like than controls while participants recover person evidence in only 3% of cases. We release 300 names with comparison controls.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 38 references that are in the map. First posted on arXiv 21 Aug 2026.
Builds on
18 of 38 references.
References · 18
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
- 2020Extracting Training Data from Large Language Modelscited by 2,189
- 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
- 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 349
- 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
- 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
- 2024Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Modelscited by 57
- 2019What's in a Name? Reducing Bias in Bios without Access to Protected Attributescited by 51
- 2025PersonaBench: Evaluating AI Models on Understanding Personal Information through Accessing (Synthetic) Private User Datacited by 27
- 2024Stop! In the Name of Flaws: Disentangling Personal Names and Sociodemographic Attributes in NLPcited by 22
- 2024JobFair: A Framework for Benchmarking Gender Hiring Bias in Large Language Modelscited by 21
6 more
- 2025Presumed Cultural Identity: How Names Shape LLM Responsescited by 21
- 2023Privacy- and Utility-Preserving NLP with Anonymized Data: A case study of Pseudonymizationcited by 18
- 2023Examining the Causal Effect of First Names on Language Models: The Case of Social Commonsense Reasoningcited by 8
- 2024ParaNames 1.0: Creating an Entity Name Corpus for 400+ Languages using Wikidatacited by 5
- 2025On the Mutual Influence of Gender and Occupation in LLM Representationscited by 4
- 2026Human-Centred LLM Privacy Audits: Findings and Frictionscited by 1
Built on by
Not cited yet.