mlsanityData to 14 Oct 2026

BERT & Family Eat Word Salad: Experiments with Text Understanding

arXiv 2101.03453 ↗First posted 10 Jan 2021cs.CL cs.AIcited by 44

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the response of large models from the BERT family to incoherent inputs that should confuse any model that claims to understand natural language. We define simple heuristics to construct such examples. Our experiments show that state-of-the-art models consistently fail to recognize them as ill-formed, and instead produce high confidence predictions on them. As a consequence of this phenomenon, models trained on sentences with randomly permuted word order perform close to state-of-the-art models. To alleviate these issues, we show that if models are explicitly trained to recognize invalid inputs, they can be robust to such attacks without a drop in performance.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 10 Jan 2021.

Builds on

32 of 44 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2017On Calibration of Modern Neural Networkscited by 4,960
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  10. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  11. 2018Deep contextualized word representationscited by 2,290
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,847
20 more
  1. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  2. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  3. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  4. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  5. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  6. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  7. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 464
  8. 2018Generating Natural Language Adversarial Examplescited by 424
  9. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  10. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  11. 2020Calibrating Deep Neural Networks using Focal Losscited by 357
  12. 2020Calibration of Pre-trained Transformerscited by 315
  13. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 305
  14. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 295
  15. 2016Enhanced LSTM for Natural Language Inferencecited by 200
  16. 2018Pathologies of Neural Models Make Interpretations Difficultcited by 140
  17. 2019Thieves on Sesame Street! Model Extraction of BERT-based APIscited by 139
  18. 2020Syntactic Data Augmentation Increases Robustness to Inference Heuristicscited by 86
  19. 2018Excessive Invariance Causes Adversarial Vulnerabilitycited by 66
  20. 2020Can Adversarial Weight Perturbations Inject Neural Backdoors?cited by 56

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 12; 2022: 13; 2023: 9; 2024: 7; 2025: 1; 2026: 2.

44 papers.

Sort:

By area:

  1. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  2. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 177
  3. 2022Better Uncertainty Calibration via Proper Scores for Classification and Beyondcited by 65
  4. 2022Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionalitycited by 44
  5. 2022Word Order Does Matter (And Shuffled Language Models Know It)cited by 32
  6. 2022When classifying grammatical role, BERT doesn't care about word order... except when it matterscited by 24
  7. 2021Rissanen Data Analysis: Examining Dataset Characteristics via Description Lengthcited by 22
  8. 2021Syntactic Perturbations Reveal Representational Correlates of Hierarchical Phrase Structure in Pretrained Language Modelscited by 14
  9. 2022Compositional Evaluation on Japanese Textual Entailment and Similaritycited by 14
  10. 2023Adversarial Clean Label Backdoor Attacks and Defenses on Text Classification Systemscited by 11
  11. 2023Consistent and Asymptotically Unbiased Estimation of Proper Calibration Errorscited by 11
  12. 2024Mitigating Reversal Curse in Large Language Models via Semantic-aware Permutation Trainingcited by 11
32 more
  1. 2021Local Structure Matters Most: Perturbation Study in NLUcited by 10
  2. 2021Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbationscited by 10
  3. 2022Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondencecited by 10
  4. 2021Studying word order through iterative shufflingcited by 9
  5. 2021Numerical reasoning in machine reading comprehension tasks: are we there yet?cited by 9
  6. 2023Don't Retrain, Just Rewrite: Countering Adversarial Perturbations by Rewriting Textcited by 8
  7. 2023Unnatural language processing: How do language models handle machine-generated prompts?cited by 8
  8. 2025Word Form Matters: LLMs' Semantic Reconstruction under Typoglycemiacited by 8
  9. 2021Schr"odinger's Tree -- On Syntax and Neural Language Modelscited by 6
  10. 2022Did AI get more negative recently?cited by 4
  11. 2023Improving Language Models Meaning Understanding and Consistency by Learning Conceptual Roles from Dictionarycited by 4
  12. 2024Enhancing Question Answering on Charts Through Effective Pre-training Taskscited by 4
  13. 2021Sometimes We Want Translationesecited by 3
  14. 2021How Does Adversarial Fine-Tuning Benefit BERT?cited by 3
  15. 2023CUE: An Uncertainty Interpretation Framework for Text Classifiers Built on Pre-Trained Language Modelscited by 3
  16. 2024Investigating the Impact of Model Instability on Explanations and Uncertaintycited by 3
  17. 2024Topic Aware Probing: From Sentence Length Prediction to Idiom Identification how reliant are Neural Language Models on Topic?cited by 3
  18. 2024Optimizing Estimators of Squared Calibration Errors in Classificationcited by 3
  19. 2021Variation and generality in encoding of syntactic anomaly information in sentence embeddingscited by 2
  20. 2022Detecting Languages Unintelligible to Multilingual Models through Local Structure Probescited by 2
  21. 2023Towards preserving word order importance through Forced Invalidationcited by 2
  22. 2022The Role of Complex NLP in Transformers for Text Ranking?cited by 1
  23. 2022Do language models make human-like predictions about the coreferents of Italian anaphoric zero pronouns?cited by 1
  24. 2022Local Structure Matters Most in Most Languagescited by 1
  25. 2023Empirical Investigation of Neural Symbolic Reasoning Strategiescited by 1
  26. 2023The Locality and Symmetry of Positional Encodingscited by 1
  27. 2024When does word order matter and when doesn't it?cited by 1
  28. 2024Word Order's Impacts: Insights from Reordering and Generation Analysiscited by 1
  29. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  30. 2022Order-sensitive Shapley Values for Evaluating Conceptual Soundness of NLP Modelscited by 0
  31. 2022Penalizing Confident Predictions on Largely Perturbed Inputs Does Not Improve Out-of-Distribution Generalization in Question Answeringcited by 0
  32. 2026Finding Sense in Nonsense with Generated Contexts: Perspectives from Humans and Language Modelscited by 0