mlsanityData to 7 Oct 2026

Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?

arXiv 2012.15180 ↗First posted 30 Dec 2020cs.CL cs.AI cs.LGcited by 72

Not on the map. Why?

Abstract (arXiv)

Do state-of-the-art natural language understanding models care about word order - one of the most important characteristics of a sequence? Not always! We found 75% to 90% of the correct predictions of BERT-based classifiers, trained on many GLUE tasks, remain constant after input words are randomly shuffled. Despite BERT embeddings are famously contextual, the contribution of each individual word to downstream tasks is almost unchanged even after the word's context is shuffled. BERT-based models are able to exploit superficial cues (e.g. the sentiment of keywords in sentiment analysis; or the word-wise similarity between sequence-pair inputs in natural language inference) to make correct decisions when tokens are arranged in random orders. Encouraging classifiers to capture word order information improves the performance on most GLUE tasks, SQuAD 2.0 and out-of-samples. Our work suggests that many GLUE tasks are not challenging machines to understand the meaning of a sentence.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 23; 2022: 20; 2023: 14; 2024: 10; 2025: 3; 2026: 2.

72 papers.

Sort:

By area:

  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  2. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  3. 2022Can language models learn from explanations in context?cited by 248
  4. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  5. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  6. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  7. 2021What Context Features Can Transformer Language Models Use?cited by 49
  8. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 48
  9. 2021Structural Persistence in Language Models: Priming as a Window into Abstract Language Representationscited by 44
  10. 2022Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionalitycited by 44
  11. 2022Contrastive language and vision learning of general fashion conceptscited by 41
  12. 2021Towards mental time travel: a hierarchical memory for reinforcement learning agentscited by 38
38 more
  1. 2022Word Order Does Matter (And Shuffled Language Models Know It)cited by 32
  2. 2024Mission: Impossible Language Modelscited by 31
  3. 2022Neural reality of argument structure constructionscited by 25
  4. 2022When classifying grammatical role, BERT doesn't care about word order... except when it matterscited by 24
  5. 2021The Case for Translation-Invariant Self-Attention in Transformer-Based Language Modelscited by 23
  6. 2021Rissanen Data Analysis: Examining Dataset Characteristics via Description Lengthcited by 22
  7. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  8. 2023Unnatural Error Correction: GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Textcited by 19
  9. 2021Syntactic Perturbations Reveal Representational Correlates of Hierarchical Phrase Structure in Pretrained Language Modelscited by 14
  10. 2021BERT might be Overkill: A Tiny but Effective Biomedical Entity Linker based on Residual Convolutional Neural Networkscited by 14
  11. 2022Compositional Evaluation on Japanese Textual Entailment and Similaritycited by 14
  12. 2024How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgmentscited by 14
  13. 2022Kernel-Whitening: Overcome Dataset Bias with Isotropic Sentence Embeddingcited by 11
  14. 2024Mitigating Reversal Curse in Large Language Models via Semantic-aware Permutation Trainingcited by 11
  15. 2021Local Structure Matters Most: Perturbation Study in NLUcited by 10
  16. 2021Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbationscited by 10
  17. 2022Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondencecited by 10
  18. 2021Studying word order through iterative shufflingcited by 9
  19. 2021Numerical reasoning in machine reading comprehension tasks: are we there yet?cited by 9
  20. 2022Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studiescited by 9
  21. 2023Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusioncited by 8
  22. 2023Unnatural language processing: How do language models handle machine-generated prompts?cited by 8
  23. 2025Word Form Matters: LLMs' Semantic Reconstruction under Typoglycemiacited by 8
  24. 2023MedGPTEval: A Dataset and Benchmark to Evaluate Responses of Large Language Models in Medicinecited by 7
  25. 2024The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewardscited by 7
  26. 2021Schr"odinger's Tree -- On Syntax and Neural Language Modelscited by 6
  27. 2024Does Incomplete Syntax Influence Korean Language Model? Focusing on Word Order and Case Markerscited by 6
  28. 2021AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defensescited by 5
  29. 2021Double Trouble: How to not explain a text classifier's decisions using counterfactuals synthesized by masked language models?cited by 5
  30. 2024Benchmark Granularity and Model Robustness for Image-Text Retrievalcited by 5
  31. 2021NLI Data Sanity Check: Assessing the Effect of Data Corruption on Model Performancecited by 4
  32. 2022Word Order Matters when you Increase Maskingcited by 4
  33. 2023The Daunting Dilemma with Sentence Encoders: Success on Standard Benchmarks, Failure in Capturing Basic Semantic Propertiescited by 4
  34. 2021Interpretable bias mitigation for textual data: Reducing gender bias in patient notes while maintaining classification performancecited by 3
  35. 2021Sometimes We Want Translationesecited by 3
  36. 2021How Does Adversarial Fine-Tuning Benefit BERT?cited by 3
  37. 2023Unsupervised Dialogue Topic Segmentation in Hyperdimensional Spacecited by 3
  38. 2023Effects of Human Adversarial and Affable Samples on BERT Generalizationcited by 3

50 most cited of 72 All 72 (JSON)