mlsanityData to 14 Oct 2026

Disentangling the Linguistic Competence of Privacy-Preserving BERT

arXiv 2310.11363 ↗First posted 17 Oct 2023cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Differential Privacy (DP) has been tailored to address the unique challenges of text-to-text privatization. However, text-to-text privatization is known for degrading the performance of language models when trained on perturbed text. Employing a series of interpretation techniques on the internal representations extracted from BERT trained on perturbed pre-text, we intend to disentangle at the linguistic level the distortion induced by differential privacy. Experimental results from a representational similarity analysis indicate that the overall similarity of internal representations is substantially reduced. Using probing tasks to unpack this dissimilarity, we find evidence that text-to-text privatization affects the linguistic competence across several formalisms, encoding localized properties of words while falling short at encoding the contextual relationships between spans of words.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 58 of its 77 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

58 of 77 references.

References · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  9. 2016Deep Learning with Differential Privacycited by 2,728
  10. 2018Deep contextualized word representationscited by 2,290
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  12. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
38 more
  1. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  2. 2019Attention is not Explanationcited by 929
  3. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  4. 2019Designing and Interpreting Probes with Control Taskscited by 640
  5. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  6. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  7. 2019Is Attention Interpretable?cited by 389
  8. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  9. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  10. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
  11. 2019Revealing the Dark Secrets of BERTcited by 301
  12. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  13. 2018Targeted Syntactic Evaluation of Language Modelscited by 242
  14. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  15. 2018Colorless green recurrent networks dream hierarchicallycited by 213
  16. 2020Information Leakage in Embedding Modelscited by 203
  17. 2019Assessing BERT's Syntactic Abilitiescited by 192
  18. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
  19. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 151
  20. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  21. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  22. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 120
  23. 2019Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbationscited by 119
  24. 2019On Identifiability in Transformerscited by 116
  25. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  26. 2017Learned in Translation: Contextualized Word Vectorscited by 104
  27. 2019The emergence of number and syntax units in LSTM language modelscited by 94
  28. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 88
  29. 2021Differential Privacy for Text Analytics via Natural Text Sanitizationcited by 77
  30. 2021Contrastive Explanations for Model Interpretabilitycited by 75
  31. 2022The Limits of Word Level Differential Privacycited by 63
  32. 2021Selective Differential Privacy for Language Modelingcited by 62
  33. 2021Conditional probing: measuring usable information beyond a baselinecited by 62
  34. 2021Natural Language Understanding with Privacy-Preserving BERTcited by 55
  35. 2018Generalised Differential Privacy for Text Document Processingcited by 52
  36. 2022A Customized Text Sanitization Mechanism with Differential Privacycited by 52
  37. 2021Probing BERT in Hyperbolic Spacescited by 50
  38. 2020A Differentially Private Text Perturbation Method Using a Regularized Mahalanobis Metriccited by 42

50 most cited of 58 All 58 (JSON)

Built on by

Not cited yet.