# On the Robustness of Language Encoders against Grammatical Errors

arXiv: https://arxiv.org/abs/2005.05683 · HTML: https://mlsanity.com/paper/2005.05683/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2020.

## Built on by (most cited first, 19 of 19)

- [On the Adversarial Robustness of Vision Transformers](https://mlsanity.com/paper/2103.15670/) (2021)
- [On the Paradox of Learning to Reason from Data](https://mlsanity.com/paper/2205.11502/) (2022)
- [Red Teaming Language Model Detectors with Language Models](https://mlsanity.com/paper/2305.19713/) (2023)
- [RuCoLA: Russian Corpus of Linguistic Acceptability](https://mlsanity.com/paper/2210.12814/) (2022)
- [Acceptability Judgements via Examining the Topology of Attention Maps](https://mlsanity.com/paper/2205.09630/) (2022)
- [Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation](https://mlsanity.com/paper/2107.11252/) (2021)
- [Backdoor Attacks on Dense Retrieval via Public and Unintentional Triggers](https://mlsanity.com/paper/2402.13532/) (2024)
- [Contrastive Fine-tuning Improves Robustness for Neural Rankers](https://mlsanity.com/paper/2105.12932/) (2021)
- [Discovering Differences in the Representation of People using Contextualized Semantic Axes](https://mlsanity.com/paper/2210.12170/) (2022)
- [Out of Style: RAG's Fragility to Linguistic Variation](https://mlsanity.com/paper/2504.08231/) (2025)
- [BERTwich: Extending BERT's Capabilities to Model Dialectal and Noisy Text](https://mlsanity.com/paper/2311.00116/) (2023)
- [We're Calling an Intervention: Exploring Fundamental Hurdles in Adapting Language Models to Nonstandard Text](https://mlsanity.com/paper/2404.07304/) (2024)
- [COBIAS: Assessing the Contextual Reliability of Bias Benchmarks for Language Models](https://mlsanity.com/paper/2402.14889/) (2024)
- [NATURE: Natural Auxiliary Text Utterances for Realistic Spoken Language Evaluation](https://mlsanity.com/paper/2111.05196/) (2021)
- [Variation and generality in encoding of syntactic anomaly information in sentence embeddings](https://mlsanity.com/paper/2111.06644/) (2021)
- [ADDMU: Detection of Far-Boundary Adversarial Examples with Data and Model Uncertainty Estimation](https://mlsanity.com/paper/2210.12396/) (2022)
- [SenTest: Evaluating Robustness of Sentence Encoders](https://mlsanity.com/paper/2311.17722/) (2023)
- [Open-DeBias: Toward Mitigating Open-Set Bias in Language Models](https://mlsanity.com/paper/2509.23805/) (2025)
- [Lost in the Request: How Communication Variation Disrupts Retrieval and Action in Email Agents](https://mlsanity.com/paper/2610.02627/) (2026)

## Abstract (arXiv)

We conduct a thorough study to diagnose the behaviors of pre-trained language encoders (ELMo, BERT, and RoBERTa) when confronted with natural grammatical errors. Specifically, we collect real grammatical errors from non-native speakers and conduct adversarial attacks to simulate these errors on clean text data. We use this approach to facilitate debugging models on downstream applications. Results confirm that the performance of all tested models is affected but the degree of impact varies. To interpret model behaviors, we further design a linguistic acceptability task to reveal their abilities in identifying ungrammatical sentences and the position of errors. We find that fixed contextual encoders with a simple classifier trained on the prediction of sentence correctness are able to locate error positions. We also design a cloze test for BERT and discover that BERT captures the interaction between errors and specific tokens in context. Our results shed light on understanding the robustness and behaviors of language encoders against grammatical errors.