# Posterior Differential Regularization with f-divergence for Improving Model Robustness

arXiv: https://arxiv.org/abs/2010.12638 · HTML: https://mlsanity.com/paper/2010.12638/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2020.

## Built on by (most cited first, 20 of 20)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations](https://mlsanity.com/paper/2306.04618/) (2023)
- [Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/2112.07869/) (2021)
- [NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned](https://mlsanity.com/paper/2101.00133/) (2021)
- [Exploring the Landscape of Machine Unlearning: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2305.06360/) (2023)
- [UnitedQA: A Hybrid Approach for Open Domain Question Answering](https://mlsanity.com/paper/2101.00178/) (2021)
- [Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention](https://mlsanity.com/paper/2112.03254/) (2021)
- [Open Domain Question Answering with A Unified Knowledge Interface](https://mlsanity.com/paper/2110.08417/) (2021)
- [ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks](https://mlsanity.com/paper/2310.12516/) (2023)
- [DIALKI: Knowledge Identification in Conversational Systems through Dialogue-Document Contextualization](https://mlsanity.com/paper/2109.04673/) (2021)
- [METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals](https://mlsanity.com/paper/2204.06644/) (2022)
- [Grow-and-Clip: Informative-yet-Concise Evidence Distillation for Answer Explanation](https://mlsanity.com/paper/2201.05088/) (2022)
- [Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledge](https://mlsanity.com/paper/2210.12338/) (2022)
- [Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding](https://mlsanity.com/paper/2110.08419/) (2021)
- [Unifying Model Explainability and Robustness for Joint Text Classification and Rationale Extraction](https://mlsanity.com/paper/2112.10424/) (2021)
- [Robust Question Answering against Distribution Shifts with Test-Time Adaptation: An Empirical Study](https://mlsanity.com/paper/2302.04618/) (2023)
- [Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach](https://mlsanity.com/paper/2104.04886/) (2021)
- [Targeted Adversarial Training for Natural Language Understanding](https://mlsanity.com/paper/2104.05847/) (2021)
- [How Does Adversarial Fine-Tuning Benefit BERT?](https://mlsanity.com/paper/2108.13602/) (2021)
- [R'enyi Neural Processes](https://mlsanity.com/paper/2405.15991/) (2024)

## Abstract (arXiv)

We address the problem of enhancing model robustness through regularization. Specifically, we focus on methods that regularize the model posterior difference between clean and noisy inputs. Theoretically, we provide a connection of two recent methods, Jacobian Regularization and Virtual Adversarial Training, under this framework. Additionally, we generalize the posterior differential regularization to the family of f-divergences and characterize the overall regularization framework in terms of Jacobian matrix. Empirically, we systematically compare those regularizations and standard BERT training on a diverse set of tasks to provide a comprehensive profile of their effect on model in-domain and out-of-domain generalization. For both fully supervised and semi-supervised settings, our experiments show that regularizing the posterior differential with f-divergence can result in well-improved model robustness. In particular, with a proper f-divergence, a BERT-base model can achieve comparable generalization as its BERT-large counterpart for in-domain, adversarial and domain shift scenarios, indicating the great potential of the proposed framework for boosting model generalization for NLP models.