# Robustifying Language Models with Test-Time Adaptation

arXiv: https://arxiv.org/abs/2310.19177 · HTML: https://mlsanity.com/paper/2310.19177/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 29 Oct 2023.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [TextBugger: Generating Adversarial Text Against Real-world Applications](https://mlsanity.com/paper/1812.05271/) (2018)
- [Counter-fitting Word Vectors to Linguistic Constraints](https://mlsanity.com/paper/1603.00892/) (2016)
- [Understanding Zero-Shot Adversarial Robustness for Large-Scale Models](https://mlsanity.com/paper/2212.07016/) (2022)
- [Adversarial Attacks are Reversible with Natural Supervision](https://mlsanity.com/paper/2103.14222/) (2021)
- [Comparing BERT against traditional machine learning text classification](https://mlsanity.com/paper/2005.13012/) (2020)
- [Robust Perception through Equivariance](https://mlsanity.com/paper/2212.06079/) (2022)
- [Using Multiple Self-Supervised Tasks Improves Model Robustness](https://mlsanity.com/paper/2204.03714/) (2022)
- [Adversarially Robust Video Perception by Seeing Motion](https://mlsanity.com/paper/2212.07815/) (2022)

## Built on by (most cited first, 2 of 2)

- [Position: Stop Acting Like Language Model Agents Are Normal Agents](https://mlsanity.com/paper/2502.10420/) (2025)
- [Agent Identity Evals: Measuring Agentic Identity](https://mlsanity.com/paper/2507.17257/) (2025)

## Abstract (arXiv)

Large-scale language models achieved state-of-the-art performance over a number of language tasks. However, they fail on adversarial language examples, which are sentences optimized to fool the language models but with similar semantic meanings for humans. While prior work focuses on making the language model robust at training time, retraining for robustness is often unrealistic for large-scale foundation models. Instead, we propose to make the language models robust at test time. By dynamically adapting the input sentence with predictions from masked words, we show that we can reverse many language adversarial attacks. Since our approach does not require any training, it works for novel tasks at test time and can adapt to novel adversarial corruptions. Visualizations and empirical results on two popular sentence classification datasets demonstrate that our method can repair adversarial language attacks over 65% o