mlsanityData to 14 Oct 2026

REDACT: A Systematically Controlled Multilingual Benchmark for Personal Information Detection

arXiv 2606.19881 ↗First posted 18 Jun 2026cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Benchmark infrastructure for personally identifiable information (PII) detection remains limited: existing corpora cover few entity types, use ad hoc generation conditions, and do not show which surface conditions cause detector failures. We present REDACT, a systematically controlled multilingual PII benchmark with 13,427 records, 324,078 entity annotations, 51 entity types, 4,127 surface-form patterns, and 25 languages across 9 scripts. A strength-2 covering-array sampler controls nine generation axes: domain, format, difficulty, length, density, code-switching, language, adjacency, and co-occurrence. Three entity-level metadata fields (disclosure status, disclosure form, and a GDPR-aligned sensitivity tier) enable stratified evaluation beyond aggregate or per-type F1. From the full benchmark, we evaluate five detectors (Presidio, GLiNER, the OpenAI Privacy Filter, GPT-4.1, and Claude Sonnet 4.6) on a locked, language-stratified sample of 1,000 records. Aggregate F1 masks an architecture-dependent failure structure: the rule-based detector performs poorly on the highest-stakes data, including HIGH-sensitivity categories (recall 0.07) and non-verbatim disclosure forms, while the LLM detectors remain more robust, with the HIGH tier as their strongest sensitivity slice. A three-model reference-free LLM-as-judge assessment corroborates that sensitivity-tier assignment is the task's hardest axis. We release the benchmark, schema, prompts, and stratified evaluation harness.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 11 of its 41 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

11 of 41 references.

References · 11

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  6. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  7. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  8. 2022The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymizationcited by 95
  9. 2023PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Modelscited by 37
  10. 2026PIIBench: A Unified Multi-Source Benchmark Corpus for Personally Identifiable Information Detectioncited by 6
  11. 2026GLiNER2-PII: A Multilingual Model for Personally Identifiable Information Extractioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026PII-TRACE: A Benchmark for Context-Aware PII Detection in Multi-Turn LLM Conversationscited by 1
  2. 2026The Text Beside the Image: Detection, Utility and Leakage for Trustworthy Multimodal Medical Data and Beyondcited by 0
  3. 2026Strong Multilingual Privacy Tagging at Encoder Speedcited by 0