mlsanityData to 7 Oct 2026

Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little

arXiv 2104.06644 ↗First posted 14 Apr 2021cs.CL cs.LGcited by 176

Not on the map. Why?

Abstract (arXiv)

A possible explanation for the impressive performance of masked language model (MLM) pre-training is that such models have learned to represent the syntactic structures prevalent in classical NLP pipelines. In this paper, we propose a different explanation: MLMs succeed on downstream tasks almost entirely due to their ability to model higher-order word co-occurrence statistics. To demonstrate this, we pre-train MLMs on sentences with randomly shuffled word order, and show that these models still achieve high accuracy after fine-tuning on many downstream tasks -- including on tasks specifically designed to be challenging for models that ignore word order. Our models perform surprisingly well according to some parametric syntactic probes, indicating possible deficiencies in how we test representations for syntactic information. Overall, our results show that purely distributional information largely explains the success of pre-training, and underscore the importance of curating challenging evaluation datasets that require deeper linguistic knowledge.

Key facts. Cited by 176 papers in the map (data through 7 Oct 2026). Builds on 61 of its 112 references that are in the map. First posted on arXiv 14 Apr 2021.

Builds on

61 of 112 references.

References · 61

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  9. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  10. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  11. 2017Deep Image Priorcited by 1,097
  12. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
38 more
  1. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  2. 2018Neural Network Acceptability Judgmentscited by 952
  3. 2019Attention is not Explanationcited by 916
  4. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  5. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  6. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  7. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  8. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  9. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  10. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  11. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  12. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  13. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  14. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  15. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  16. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  17. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  18. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  19. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  20. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  21. 2019Assessing BERT's Syntactic Abilitiescited by 191
  22. 2018Multilingual Constituency Parsing with Self-Attention and Pre-Trainingcited by 117
  23. 2019Exploring Randomly Wired Neural Networks for Image Recognitioncited by 103
  24. 2020Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNscited by 103
  25. 2021Pre-training without Natural Imagescited by 100
  26. 2018Transforming Question Answering Datasets Into Natural Language Inference Datasetscited by 99
  27. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  28. 2019The emergence of number and syntax units in LSTM language modelscited by 92
  29. 2020OCNLI: Original Chinese Natural Language Inferencecited by 86
  30. 2020Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?cited by 72
  31. 2020UnNatural Language Inferencecited by 68
  32. 2018Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailmentcited by 58
  33. 2019Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIscited by 58
  34. 2019Learning to Few-Shot Learn Across Diverse Natural Language Classification Taskscited by 57
  35. 2020Pareto Probing: Trading Off Accuracy for Complexitycited by 55
  36. 2020Efficient Second-Order TreeCRF for Neural Dependency Parsingcited by 48
  37. 2021BERT & Family Eat Word Salad: Experiments with Text Understandingcited by 44
  38. 2020Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humanscited by 42

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 32; 2022: 50; 2023: 36; 2024: 32; 2025: 19; 2026: 7.

176 papers.

Sort:

By area:

  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  2. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  3. 2021True Few-Shot Learning with Language Modelscited by 343
  4. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
  5. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  6. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  7. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  8. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  9. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  10. 2023System 2 Attention (is something you might need too)cited by 85
  11. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  12. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
38 more
  1. 2024Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNscited by 65
  2. 2022Probing for the Usage of Grammatical Numbercited by 57
  3. 2023Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generationcited by 56
  4. 2023Revisiting the Role of Language Priors in Vision-Language Modelscited by 50
  5. 2024Reverse Training to Nurse the Reversal Cursecited by 48
  6. 2021Structural Persistence in Language Models: Priming as a Window into Abstract Language Representationscited by 44
  7. 2022Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionalitycited by 44
  8. 2023RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressionscited by 43
  9. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Modelscited by 43
  10. 2024Changing Answer Order Can Decrease MMLU Accuracycited by 41
  11. 2024Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized Reasoning Tracescited by 40
  12. 2022Word Order Does Matter (And Shuffled Language Models Know It)cited by 32
  13. 2024Mission: Impossible Language Modelscited by 31
  14. 2021Does Pretraining for Summarization Require Knowledge Transfer?cited by 30
  15. 2021How BPE Affects Memorization in Transformerscited by 29
  16. 2022Pretraining with Artificial Language: Studying Transferable Knowledge in Language Modelscited by 28
  17. 2023A Better Way to Do Masked Language Model Scoringcited by 27
  18. 2021On the Transferability of Pre-trained Language Models: A Study from Artificial Datasetscited by 25
  19. 2022Insights into Pre-training via Simpler Synthetic Taskscited by 25
  20. 2022ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metricscited by 25
  21. 2024The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and Morecited by 25
  22. 2022When classifying grammatical role, BERT doesn't care about word order... except when it matterscited by 24
  23. 2022Position Prediction as an Effective Pretraining Strategycited by 24
  24. 2021The Case for Translation-Invariant Self-Attention in Transformer-Based Language Modelscited by 23
  25. 2022Unit Testing for Concepts in Neural Networkscited by 23
  26. 2022When are Lemons Purple? The Concept Association Bias of Vision-Language Modelscited by 23
  27. 2023Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddingscited by 23
  28. 2021Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identitycited by 22
  29. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  30. 2022Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?cited by 20
  31. 2022Downstream Datasets Make Surprisingly Good Pretraining Corporacited by 20
  32. 2022Processing Long Legal Documents with Pre-trained Transformers: Modding LegalBERT and Longformercited by 20
  33. 2023A 5' UTR Language Model for Decoding Untranslated Regions of mRNA and Function Predictionscited by 20
  34. 2021A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervisioncited by 19
  35. 2023Unnatural Error Correction: GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Textcited by 19
  36. 2024Fairness and Bias in Multimodal AI: A Surveycited by 19
  37. 2022LAD: Language Models as Data for Zero-Shot Dialogcited by 18
  38. 2024T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretrainingcited by 18

50 most cited of 176 All 176 (JSON)