mlsanityData to 14 Oct 2026

Distilling Human-Aligned Privacy Sensitivity Assessment from Large Language Models

arXiv 2603.29497 ↗First posted 31 Mar 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Accurate privacy evaluation of textual data remains a critical challenge in privacy-preserving natural language processing. Recent work has shown that large language models (LLMs) can serve as reliable privacy evaluators, achieving strong agreement with human judgments; however, their computational cost and impracticality for processing sensitive data at scale limit real-world deployment. We address this gap by distilling the privacy assessment capabilities of Mistral Large 3 (675B) into lightweight encoder models with as few as 150M parameters. Leveraging a large-scale dataset of privacy-annotated texts spanning 10 diverse domains, we train efficient classifiers that preserve strong agreement with human annotations while dramatically reducing computational requirements. We validate our approach on human-annotated test data and demonstrate its practical utility as an evaluation metric for de-identification systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 29 references that are in the map. First posted on arXiv 31 Mar 2026.

Builds on

14 of 29 references.

References · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  6. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  7. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  8. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 116
  9. 2022The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymizationcited by 95
  10. 2024Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wildcited by 47
  11. 2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 38
  12. 2025LLM-as-a-Judge for Privacy Evaluation? Exploring the Alignment of Human and LLM Perceptions of Privacy in Textual Datacited by 10
2 more
  1. 2025How do we measure privacy in text? A survey of text anonymization metricscited by 5
  2. 2025Tau-Eval: A Unified Evaluation Framework for Useful and Private Text Anonymizationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Privacy-Preserving Large Language Models: Mechanisms, Applications, and Future Directions
  2. 2024Preserving Privacy in Large Language Models: A Survey on Current Threats and Solutions
  3. 2025A Survey on Privacy Risks and Protection in Large Language Models
  4. 2026Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptation
  5. 2024PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restoration
  6. 2026LLM-CEG: Extending the Classification Error Gauge Framework for Privacy Auditing of Large Language Models ↗
  7. 2025PrivacyScalpel: Enhancing LLM Privacy via Interpretable Feature Intervention with Sparse Autoencoders
  8. 2025Privacy Auditing of Large Language Models