mlsanityData to 7 Oct 2026

Robust Natural Language Understanding with Residual Attention Debiasing

arXiv 2305.17627 ↗First posted 28 May 2023cs.CL cs.AI cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Natural language understanding (NLU) models often suffer from unintended dataset biases. Among bias mitigation methods, ensemble-based debiasing methods, especially product-of-experts (PoE), have stood out for their impressive empirical success. However, previous ensemble-based debiasing methods typically apply debiasing on top-level logits without directly addressing biased attention patterns. Attention serves as the main media of feature interaction and aggregation in PLMs and plays a crucial role in providing robust prediction. In this paper, we propose REsidual Attention Debiasing (READ), an end-to-end debiasing method that mitigates unintended biases from attention. Experiments on three NLU tasks show that READ significantly improves the performance of BERT-based models on OOD data with shortcuts removed, including +12.9% accuracy on HANS, +11.0% accuracy on FEVER-Symmetric, and +2.7% F1 on PAWS. Detailed analyses demonstrate the crucial role of unbiased attention in robust NLU models and that READ effectively mitigates biases in attention. Code is available at https://github.com/luka-group/READ.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 61 of its 74 references that are in the map. First posted on arXiv 28 May 2023.

Builds on

61 of 74 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  11. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  12. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
38 more
  1. 2019Attention is not Explanationcited by 916
  2. 2018Robustness May Be at Odds with Accuracycited by 873
  3. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  4. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  5. 2019Attention is not not Explanationcited by 629
  6. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  7. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  8. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  9. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  10. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  11. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  12. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
  13. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  14. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  15. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
  16. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  17. 2016Understanding Neural Networks through Representation Erasurecited by 223
  18. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  19. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  20. 2019Certified Robustness to Adversarial Word Substitutionscited by 164
  21. 2020Understanding and Mitigating the Tradeoff Between Robustness and Accuracycited by 148
  22. 2019Towards Debiasing Fact Verification Modelscited by 143
  23. 2019End-to-End Bias Mitigation by Modelling Biases in Corporacited by 127
  24. 2019Unlearn Dataset Bias in Natural Language Inference by Fitting the Residualcited by 119
  25. 2020Towards Debiasing NLU Models from Unknown Biasescited by 116
  26. 2019Learning to Deceive with Attention-Based Explanationscited by 111
  27. 2021Competency Problems: On Finding and Removing Artifacts in Language Datacited by 91
  28. 2020Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performancecited by 85
  29. 2020Learning from others' mistakes: Avoiding dataset biases without modeling themcited by 83
  30. 2021Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Modelscited by 78
  31. 2022Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Listscited by 54
  32. 2018A Unified Model for Extractive and Abstractive Summarization using Inconsistency Losscited by 47
  33. 2021SparseBERT: Rethinking the Importance Analysis in Self-attentioncited by 46
  34. 2020Universal Adversarial Attack on Attention and the Resulting Dataset DAmageNetcited by 45
  35. 2022Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformercited by 45
  36. 2021Prevent the Language Model from being Overconfident in Neural Machine Translationcited by 34
  37. 2022Robust (Controlled) Table-to-Text Generation with Structure-Aware Equivariance Learningcited by 32
  38. 2022Should We Rely on Entity Mentions for Relation Extraction? Debiasing Relation Extraction with Counterfactual Analysiscited by 30

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 3; 2026: 2.

10 papers.

  1. 2023On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlookcited by 15
  2. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 14
  3. 2024Assessing LLMs for Zero-shot Abstractive Summarization Through the Lens of Relevance Paraphrasingcited by 11
  4. 2024Two Heads are Better than One: Nested PoE for Robust Defense Against Multi-Backdoorscited by 6
  5. 2025Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selectioncited by 4
  6. 2024Towards Better Graph-based Cross-document Relation Extraction via Non-bridge Entity Enhancement and Prediction Debiasingcited by 3
  7. 2025Do Students Debias Like Teachers? On the Distillability of Bias Mitigation Methodscited by 2
  8. 2025FairFlow: Mitigating Dataset Biases through Undecided Learningcited by 0
  9. 2026Efficient and Interpretable Transformer for Counterfactual Fairnesscited by 0
  10. 2026Fair Cognitive Impairment Detection Through Unlearningcited by 0