mlsanityData to 13 Oct 2026

Local Structure Matters Most: Perturbation Study in NLU

arXiv 2107.13955 ↗First posted 29 Jul 2021cs.CL cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Recent research analyzing the sensitivity of natural language understanding models to word-order perturbations has shown that neural models are surprisingly insensitive to the order of words. In this paper, we investigate this phenomenon by developing order-altering perturbations on the order of words, subwords, and characters to analyze their effect on neural models' performance on language understanding tasks. We experiment with measuring the impact of perturbations to the local neighborhood of characters and global position of characters in the perturbed texts and observe that perturbation functions found in prior literature only affect the global ordering while the local ordering remains relatively unperturbed. We empirically show that neural models, invariant of their inductive biases, pretraining scheme, or the choice of tokenization, mostly rely on the local structure of text to build understanding and make limited use of the global structure.

Key facts. Cited by 10 papers in the map (data through 13 Oct 2026). Builds on 41 of its 74 references that are in the map. First posted on arXiv 29 Jul 2021.

Builds on

41 of 74 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,480
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,880
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,133
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,910
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,676
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,215
  12. 2016Enriching Word Vectors with Subword Informationcited by 1,581
29 more
  1. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,200
  2. 2018Neural Network Acceptability Judgmentscited by 957
  3. 2020A Primer in BERTology: What we know about how BERT workscited by 809
  4. 2014Distributed Representations of Sentences and Documentscited by 677
  5. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 507
  6. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 437
  7. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  8. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 344
  9. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  10. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
  11. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 231
  12. 2018Colorless green recurrent networks dream hierarchicallycited by 211
  13. 2019Assessing BERT's Syntactic Abilitiescited by 191
  14. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  15. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 160
  16. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
  17. 2020Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?cited by 72
  18. 2020UnNatural Language Inferencecited by 68
  19. 2020CharBERT: Character-aware Pre-trained Language Modelcited by 63
  20. 2020Vocabulary Learning via Optimal Transport for Neural Machine Translationcited by 58
  21. 2021What Context Features Can Transformer Language Models Use?cited by 49
  22. 2019Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Studycited by 47
  23. 2021BERT & Family Eat Word Salad: Experiments with Text Understandingcited by 44
  24. 2020Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigationcited by 34
  25. 2019Correlating neural and symbolic representations of languagecited by 33
  26. 2019Improving Zero-shot Translation with Language-Independent Constraintscited by 33
  27. 2019Improving Pre-Trained Multilingual Models with Vocabulary Expansioncited by 27
  28. 2015Self-Adaptive Hierarchical Sentence Modelcited by 10
  29. 2021Sometimes We Want Translationesecited by 3

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 2; 2022: 2; 2023: 1; 2024: 3; 2025: 1; 2026: 1.

10 papers.

  1. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 156
  2. 2024Interpretability Needs a New Paradigmcited by 11
  3. 2021Schr"odinger's Tree -- On Syntax and Neural Language Modelscited by 6
  4. 2025Explainable Mapper: Charting LLM Embedding Spaces Using Perturbation-Based Explanation and Verification Agentscited by 4
  5. 2022Detecting Languages Unintelligible to Multilingual Models through Local Structure Probescited by 2
  6. 2022Local Structure Matters Most in Most Languagescited by 1
  7. 2023The Locality and Symmetry of Positional Encodingscited by 1
  8. 2024When does word order matter and when doesn't it?cited by 1
  9. 2024Word Order's Impacts: Insights from Reordering and Generation Analysiscited by 1
  10. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1