# Scalable multilingual PII annotation for responsible AI in LLMs

arXiv: https://arxiv.org/abs/2510.06250 · HTML: https://mlsanity.com/paper/2510.06250/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 29 references that are in the map.
- First posted on arXiv 3 Oct 2025.

## Builds on (most cited first, 8 of 8)

- [A Survey on Deep Learning for Named Entity Recognition](https://mlsanity.com/paper/1812.09449/) (2018)
- [Variable Rate Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06085/) (2015)
- [MasakhaNER: Named Entity Recognition for African Languages](https://mlsanity.com/paper/2103.11811/) (2021)
- [De-identification of Patient Notes with Recurrent Neural Networks](https://mlsanity.com/paper/1606.03475/) (2016)
- [Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages](https://mlsanity.com/paper/2305.17179/) (2023)
- [Analyzing Dataset Annotation Quality Management in the Wild](https://mlsanity.com/paper/2307.08153/) (2023)
- [Automated Privacy Information Annotation in Large Language Model Interactions](https://mlsanity.com/paper/2505.20910/) (2025)
- [Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts](https://mlsanity.com/paper/2204.04775/) (2022)

## Built on by (most cited first, 2 of 2)

- [DialogPII: A multilingual dataset of synthetic dialog transcripts to detect personal information](https://mlsanity.com/paper/2606.30312/) (2026)
- [Addressing Labelled Data Scarcity: Taxonomy-Agnostic Annotation of PII Values in HTTP Traffic using LLMs](https://mlsanity.com/paper/2605.06305/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) gain wider adoption, ensuring their reliable handling of Personally Identifiable Information (PII) across diverse regulatory contexts has become essential. This work introduces a scalable multilingual data curation framework designed for high-quality PII annotation across 13 underrepresented locales, covering approximately 336 locale-specific PII types. Our phased, human-in-the-loop annotation methodology combines linguistic expertise with rigorous quality assurance, leading to substantial improvements in recall and false positive rates from pilot, training, and production phases. By leveraging inter-annotator agreement metrics and root-cause analysis, the framework systematically uncovers and resolves annotation inconsistencies, resulting in high-fidelity datasets suitable for supervised LLM fine-tuning. Beyond reporting empirical gains, we highlight common annotator challenges in multilingual PII labeling and demonstrate how iterative, analytics-driven pipelines can enhance both annotation quality and downstream model reliability.