mlsanityData to 14 Oct 2026

Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

arXiv 2110.01518 ↗First posted 4 Oct 2021cs.CLcited by 67

Not on the map. Why?

Abstract (arXiv)

Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Builds on 33 of its 45 references that are in the map. First posted on arXiv 4 Oct 2021.

Builds on

33 of 45 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  10. 2016Deep Variational Information Bottleneckcited by 930
  11. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  12. 2020A Primer in BERTology: What we know about how BERT workscited by 815
21 more
  1. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  2. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  3. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  4. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  5. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  6. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 295
  7. 2016Enhanced LSTM for Natural Language Inferencecited by 200
  8. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 175
  9. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  10. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 134
  11. 2019Learning Robust Representations by Projecting Superficial Statistics Outcited by 123
  12. 2020Towards Debiasing NLU Models from Unknown Biasescited by 116
  13. 2021Competency Problems: On Finding and Removing Artifacts in Language Datacited by 91
  14. 2019BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performancecited by 89
  15. 2020Syntactic Data Augmentation Increases Robustness to Inference Heuristicscited by 86
  16. 2020Learning from others' mistakes: Avoiding dataset biases without modeling themcited by 83
  17. 2021Variational Information Bottleneck for Effective Low-Resource Fine-Tuningcited by 51
  18. 2019Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inferencecited by 49
  19. 2020Learning to Model and Ignore Dataset Bias with Mixed Capacity Ensemblescited by 43
  20. 2020Towards Robustifying NLI Models Against Lexical Dataset Biasescited by 34
  21. 2020Improving Robustness by Augmenting Training Sentences with Predicate-Argument Structurescited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 10; 2023: 15; 2024: 22; 2025: 11; 2026: 7.

67 papers.

Sort:

By area:

  1. 2023Protein Design with Guided Discrete Diffusioncited by 157
  2. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  3. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 85
  4. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  5. 2023DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Modelscited by 79
  6. 2022Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasetscited by 54
  7. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  8. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
  9. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Expertscited by 37
  10. 2022Text Revealer: Private Text Reconstruction via Model Inversion Attacks against Transformerscited by 31
  11. 2022An Efficiency Study for SPLADE Modelscited by 26
  12. 2023Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicinecited by 24
38 more
  1. 2022Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustnesscited by 23
  2. 2023LLMs Can Understand Encrypted Prompt: Towards Privacy-Computing Friendly Transformerscited by 22
  3. 2023Position: Key Claims in LLM Research Have a Long Tail of Footnotescited by 20
  4. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 20
  5. 2021Pro-KD: Progressive Distillation by Following the Footsteps of the Teachercited by 17
  6. 2026Calibrate-Then-Act: Cost-Aware Exploration in LLM Agentscited by 17
  7. 2023CLIPTER: Looking at the Bigger Picture in Scene Text Recognitioncited by 16
  8. 2022Machine Reading, Fast and Slow: When Do Models "Understand" Language?cited by 13
  9. 2024FedFa: A Fully Asynchronous Training Paradigm for Federated Learningcited by 12
  10. 2024Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Modelscited by 11
  11. 2024Context-Guided Diffusion for Out-of-Distribution Molecular and Protein Designcited by 11
  12. 2024Unified Generation, Reconstruction, and Representation: Generalized Diffusion with Adaptive Latent Encoding-Decodingcited by 10
  13. 2024The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Modelscited by 10
  14. 2021PECO: Examining Single Sentence Label Leakage in Natural Language Inference Datasets through Progressive Evaluation of Cluster Outlierscited by 8
  15. 2023Web Content Filtering through knowledge distillation of Large Language Modelscited by 7
  16. 2024CTPD: Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysiscited by 6
  17. 2025Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stabilitycited by 6
  18. 2024Understanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERTcited by 5
  19. 2024Exploring Continual Learning of Compositional Generalization in NLIcited by 4
  20. 2024Shallow Cross-Encoders for Low-Latency Retrievalcited by 4
  21. 2025To Judge or not to Judge: Using LLM Judgements for Advertiser Keyphrase Relevance at eBaycited by 4
  22. 2022Looking at the Overlooked: An Analysis on the Word-Overlap Bias in Natural Language Inferencecited by 3
  23. 2023Measuring Feature Sparsity in Language Modelscited by 3
  24. 2024T-Rex: Text-assisted Retrosynthesis Predictioncited by 3
  25. 2024MCUBERT: Memory-Efficient BERT Inference on Commodity Microcontrollerscited by 3
  26. 2025DACTYL: Diverse Adversarial Corpus of Texts Yielded from Large Language Modelscited by 3
  27. 2026Cross-Domain Transfer with Particle Physics Foundation Models: From Jets to Neutrino Interactionscited by 3
  28. 2024Open foundation models for Azerbaijani languagecited by 2
  29. 2024Auto-GDA: Automatic Domain Adaptation for Efficient Grounding Verification in Retrieval-Augmented Generationcited by 2
  30. 2025Revealing Hidden Mechanisms of Cross-Country Content Moderation with Natural Language Processingcited by 2
  31. 2022AGO: Boosting Mobile AI Inference Performance by Removing Constraints on Graph Optimizationcited by 1
  32. 2024Natural Language Processing and Multimodal Stock Price Predictioncited by 1
  33. 2024Exploring Domain Robust Lightweight Reward Models based on Router Mechanismcited by 1
  34. 2023CMLCompiler: A Unified Compiler for Classical Machine Learningcited by 0
  35. 2023CoSiNES: Contrastive Siamese Network for Entity Standardizationcited by 0
  36. 2023WIKITIDE: A Wikipedia-Based Timestamped Definition Pairs Datasetcited by 0
  37. 2023Latent Feature-based Data Splits to Improve Generalisation Evaluation: A Hate Speech Detection Case Studycited by 0
  38. 2024Advancing NLP Models with Strategic Text Augmentation: A Comprehensive Study of Augmentation Methods and Curriculum Strategiescited by 0

50 most cited of 67 All 67 (JSON)