# IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Method

arXiv: https://arxiv.org/abs/2505.06889 · HTML: https://mlsanity.com/paper/2505.06889/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 48 references that are in the map.
- First posted on arXiv 11 May 2025.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [FreeLB: Enhanced Adversarial Training for Natural Language Understanding](https://mlsanity.com/paper/1909.11764/) (2019)
- [SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization](https://mlsanity.com/paper/1911.03437/) (2019)
- [Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models](https://mlsanity.com/paper/2111.02840/) (2021)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [Adversarial Training and Robustness for Multiple Perturbations](https://mlsanity.com/paper/1904.13000/) (2019)
- [Better Fine-Tuning by Reducing Representational Collapse](https://mlsanity.com/paper/2008.03156/) (2020)
- [Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View](https://mlsanity.com/paper/1906.02762/) (2019)
- [Recurrent Neural Networks (RNNs): A gentle Introduction and Overview](https://mlsanity.com/paper/1912.05911/) (2019)
- [PolyNet: A Pursuit of Structural Diversity in Very Deep Networks](https://mlsanity.com/paper/1611.05725/) (2016)
- [On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers](https://mlsanity.com/paper/2102.07346/) (2021)
- [Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models](https://mlsanity.com/paper/2105.11136/) (2021)
- [Bridge the Gap Between CV and NLP! A Gradient-based Textual Adversarial Attack Framework](https://mlsanity.com/paper/2110.15317/) (2021)
- [ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation](https://mlsanity.com/paper/2203.09176/) (2022)
- [Towards Robust ResNet: A Small Step but A Giant Leap](https://mlsanity.com/paper/1902.10887/) (2019)
- [N-ODE Transformer: A Depth-Adaptive Variant of the Transformer Using Neural Ordinary Differential Equations](https://mlsanity.com/paper/2010.11358/) (2020)
- [Robust learning with implicit residual networks](https://mlsanity.com/paper/1905.10479/) (2019)
- [Implicit Euler ODE Networks for Single-Image Dehazing](https://mlsanity.com/paper/2007.06443/) (2020)

## Built on by (most cited first, 3 of 3)

- [WARP: Guaranteed Inner-Layer Repair of NLP Transformers](https://mlsanity.com/paper/2604.00938/) (2026)
- [IIET: Efficient Numerical Transformer via Implicit Iterative Euler Method](https://mlsanity.com/paper/2509.22463/) (2025)
- [A Unified BERT-CNN-BiLSTM Framework for Simultaneous Headline Classification and Sentiment Analysis of Bangla News](https://mlsanity.com/paper/2511.18618/) (2025)

## Abstract (arXiv)

Pre-trained Language Models (PLMs) have achieved remarkable performance on diverse NLP tasks through pre-training and fine-tuning. However, fine-tuning the model with a large number of parameters on limited downstream datasets often leads to vulnerability to adversarial attacks, causing overfitting of the model on standard datasets. To address these issues, we propose IM-BERT from the perspective of a dynamic system by conceptualizing a layer of BERT as a solution of Ordinary Differential Equations (ODEs). Under the situation of initial value perturbation, we analyze the numerical stability of two main numerical ODE solvers: the explicit and implicit Euler approaches. Based on these analyses, we introduce a numerically robust IM-connection incorporating BERT's layers. This strategy enhances the robustness of PLMs against adversarial attacks, even in low-resource scenarios, without introducing additional parameters or adversarial training strategies. Experimental results on the adversarial GLUE (AdvGLUE) dataset validate the robustness of IM-BERT under various conditions. Compared to the original BERT, IM-BERT exhibits a performance improvement of approximately 8.3%p on the AdvGLUE dataset. Furthermore, in low-resource scenarios, IM-BERT outperforms BERT by achieving 5.9%p higher accuracy.