mlsanityData to 7 Oct 2026

End-to-End Self-Debiasing Framework for Robust NLU Training

arXiv 2109.02071 ↗First posted 5 Sep 2021cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

Existing Natural Language Understanding (NLU) models have been shown to incorporate dataset biases leading to strong performance on in-distribution (ID) test sets but poor performance on out-of-distribution (OOD) ones. We introduce a simple yet effective debiasing framework whereby the shallow representations of the main model are used to derive a bias model and both models are trained simultaneously. We demonstrate on three well studied NLU tasks that despite its simplicity, our method leads to competitive OOD results. It significantly outperforms other debiasing approaches on two tasks, while still delivering high in-distribution performance.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 29 of its 35 references that are in the map. First posted on arXiv 5 Sep 2021.

Builds on

29 of 35 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  9. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  10. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  11. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  12. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
17 more
  1. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  2. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  3. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
  4. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  5. 2019Towards Debiasing Fact Verification Modelscited by 143
  6. 2019Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overviewcited by 137
  7. 2019End-to-End Bias Mitigation by Modelling Biases in Corporacited by 127
  8. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  9. 2019Unlearn Dataset Bias in Natural Language Inference by Fitting the Residualcited by 119
  10. 2020Towards Debiasing NLU Models from Unknown Biasescited by 116
  11. 2020Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performancecited by 85
  12. 2020Learning from others' mistakes: Avoiding dataset biases without modeling themcited by 83
  13. 2020Learning to Model and Ignore Dataset Bias with Mixed Capacity Ensemblescited by 43
  14. 2018Evaluating Compositionality in Sentence Embeddingscited by 35
  15. 2019Adversarial Regularization for Visual Question Answering: Strengths, Shortcomings, and Side Effectscited by 25
  16. 2021Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognitioncited by 19
  17. 2020Improving Robustness by Augmenting Training Sentences with Predicate-Argument Structurescited by 10

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 4; 2023: 7; 2024: 5; 2025: 5.

24 papers.

Sort:

By area:

  1. 2022Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasetscited by 54
  2. 2023From Shortcuts to Triggers: Backdoor Defense with Denoised PoEcited by 26
  3. 2024LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMscited by 24
  4. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 18
  5. 2022Feature-Level Debiased Natural Language Understandingcited by 10
  6. 2023Robust Natural Language Understanding with Residual Attention Debiasingcited by 10
  7. 2022A Win-win Deal: Towards Sparse and Robust Pre-trained Language Modelscited by 7
  8. 2025EqualizeIR: Mitigating Linguistic Biases in Retrieval Modelscited by 5
  9. 2025Adversarial Attacks Against Automated Fact-Checking: A Surveycited by 5
  10. 2021How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understandingcited by 4
  11. 2022A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehensioncited by 4
  12. 2023DialogVCS: Robust Natural Language Understanding in Dialogue System Upgradecited by 4
12 more
  1. 2023A Simple yet Effective Self-Debiasing Framework for Transformer Modelscited by 4
  2. 2025Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selectioncited by 4
  3. 2024Explanation based Bias Decoupling Regularization for Natural Language Inferencecited by 3
  4. 2023Guide the Learner: Controlling Product of Experts Debiasing Method Based on Token Attribution Similaritiescited by 2
  5. 2023Improving Bias Mitigation through Bias Experts in Natural Language Understandingcited by 2
  6. 2025Mitigating Shortcut Learning with InterpoLated Learningcited by 2
  7. 2021Knowledge Distillation with Noisy Labels for Natural Language Understandingcited by 1
  8. 2023Improving Classifier Robustness through Active Generation of Pairwise Counterfactualscited by 1
  9. 2024Self-Supervised Position Debiasing for Large Language Modelscited by 1
  10. 2024Not Eliminate but Aggregate: Post-Hoc Control over Mixture-of-Experts to Address Shortcut Shifts in Natural Language Understandingcited by 0
  11. 2024Towards Robust Extractive Question Answering Models: Rethinking the Training Methodologycited by 0
  12. 2025FairFlow: Mitigating Dataset Biases through Undecided Learningcited by 0