mlsanityData to 7 Oct 2026

Granularity is crucial when applying differential privacy to text: An investigation for neural machine translation

arXiv 2407.18789 ↗First posted 26 Jul 2024cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Applying differential privacy (DP) by means of the DP-SGD algorithm to protect individual data points during training is becoming increasingly popular in NLP. However, the choice of granularity at which DP is applied is often neglected. For example, neural machine translation (NMT) typically operates on the sentence-level granularity. From the perspective of DP, this setup assumes that each sentence belongs to a single person and any two sentences in the training dataset are independent. This assumption is however violated in many real-world NMT datasets, e.g., those including dialogues. For proper application of DP we thus must shift from sentences to entire documents. In this paper, we investigate NMT at both the sentence and document levels, analyzing the privacy/utility trade-off for both scenarios, and evaluating the risks of not using the appropriate privacy granularity in terms of leaking personally identifiable information (PII). Our findings indicate that the document-level NMT system is more resistant to membership inference attacks, emphasizing the significance of using the appropriate granularity when working with DP.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 53 references that are in the map. First posted on arXiv 26 Jul 2024.

Builds on

22 of 53 references.

References · 22

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2016Deep Learning with Differential Privacycited by 2,682
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  10. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  11. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  12. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
10 more
  1. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  2. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  3. 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
  4. 2019Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?cited by 68
  5. 2021Measuring and Increasing Context Usage in Context-Aware Machine Translationcited by 48
  6. 2023Document Flattening: Beyond Concatenating Context for Document-Level Neural Machine Translationcited by 15
  7. 2020Designing the Business Conversation Corpuscited by 14
  8. 2023Identifying Context-Dependent Translations for Evaluation Set Productioncited by 14
  9. 2023mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequencescited by 10
  10. 2023DP-NMT: Scalable Differentially-Private Machine Translationcited by 7

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025RL-Finetuned LLMs for Privacy-Preserving Synthetic Rewritingcited by 0