mlsanityData to 14 Oct 2026

Can Pretrained Language Models Derive Correct Semantics from Corrupt Subwords under Noise?

arXiv 2306.15268 ↗First posted 27 Jun 2023cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

For Pretrained Language Models (PLMs), their susceptibility to noise has recently been linked to subword segmentation. However, it is unclear which aspects of segmentation affect their understanding. This study assesses the robustness of PLMs against various disrupted segmentation caused by noise. An evaluation framework for subword segmentation, named Contrastive Lexical Semantic (CoLeS) probe, is proposed. It provides a systematic categorization of segmentation corruption under noise and evaluation protocols by generating contrastive datasets with canonical-noisy word pairs. Experimental results indicate that PLMs are unable to accurately compute word meanings if the noise introduces completely different subwords, small subword fragments, or a large number of additional subwords, particularly when they are inserted within other subwords.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 33 of its 65 references that are in the map. First posted on arXiv 27 Jun 2023.

Builds on

33 of 65 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  9. 2016Deep Learning with Differential Privacycited by 2,728
  10. 2020Extracting Training Data from Large Language Modelscited by 2,189
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  12. 2018Neural Network Acceptability Judgmentscited by 958
21 more
  1. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  2. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  3. 2018Generating Natural Language Adversarial Examplescited by 424
  4. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
  5. 2016Bidirectional Attention Flow for Machine Comprehensioncited by 278
  6. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  7. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  8. 2021Unlearnable Examples: Making Personal Data Unexploitablecited by 193
  9. 2017Learning to Ask: Neural Question Generation for Reading Comprehensioncited by 186
  10. 2016Counter-fitting Word Vectors to Linguistic Constraintscited by 140
  11. 2020MetaPoison: Practical General-purpose Clean-label Data Poisoningcited by 120
  12. 2019Unlearn Dataset Bias in Natural Language Inference by Fitting the Residualcited by 119
  13. 2021LowKey: Leveraging Adversarial Attacks to Protect Social Media Users from Facial Recognitioncited by 94
  14. 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
  15. 2020Imitation Attacks and Defenses for Black-box Machine Translation Systemscited by 81
  16. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  17. 2021Why Machine Reading Comprehension Models Learn Shortcuts?cited by 40
  18. 2022Character-level White-Box Adversarial Attacks against Transformers via Attachable Subwords Substitutioncited by 28
  19. 2019GitHub Typo Corpus: A Large-Scale Multilingual Dataset of Misspellings and Grammatical Errorscited by 22
  20. 2022CharacterBERT and Self-Teaching for Improving the Robustness of Dense Retrievers on Queries with Typoscited by 22
  21. 2015Twitter Sentiment Analysis: Lexicon Method, Machine Learning Method and Their Combinationcited by 11

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 4.

6 papers.

  1. 2023Making Harmful Behaviors Unlearnable for Large Language Modelscited by 20
  2. 2023MultiDelete for Multimodal Machine Unlearningcited by 20
  3. 2024MU-Bench: A Multitask Multimodal Benchmark for Machine Unlearningcited by 17
  4. 2024Toward Availability Attacks in 3D Point Cloudscited by 7
  5. 2024HiddenSpeaker: Generate Imperceptible Unlearnable Audios for Speaker Verification Systemcited by 4
  6. 2024Scrambled text: training Language Models to correct OCR errors using synthetic datacited by 2