mlsanityData to 7 Oct 2026

Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment

arXiv 1804.08117 ↗First posted 22 Apr 2018cs.CL cs.AIcited by 58

Not on the map. Why?

Abstract (arXiv)

The quality of training data is one of the crucial problems when a learning-centered approach is employed. This paper proposes a new method to investigate the quality of a large corpus designed for the recognizing textual entailment (RTE) task. The proposed method, which is inspired by a statistical hypothesis test, consists of two phases: the first phase is to introduce the predictability of textual entailment labels as a null hypothesis which is extremely unacceptable if a target corpus has no hidden bias, and the second phase is to test the null hypothesis using a Naive Bayes model. The experimental result of the Stanford Natural Language Inference (SNLI) corpus does not reject the null hypothesis. Therefore, it indicates that the SNLI corpus has a hidden bias which allows prediction of textual entailment labels from hypothesis sentences even if no context information is given by a premise sentence. This paper also presents the performance impact of NN models for RTE caused by this hidden bias.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 23; 2022: 14; 2023: 10; 2024: 6; 2025: 5.

58 papers.

Sort:

By area:

  1. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  2. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  3. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 222
  4. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  5. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  6. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  7. 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
  8. 2022DataPerf: Benchmarks for Data-Centric AI Developmentcited by 89
  9. 2021DocNLI: A Large-scale Dataset for Document-level Natural Language Inferencecited by 75
  10. 2023Natural Language Reasoning, A Surveycited by 61
  11. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  12. 2021Fool Me Twice: Entailment from Wikipedia Gamificationcited by 50
38 more
  1. 2022Don't Blame the Annotator: Bias Already Starts in the Annotation Instructionscited by 47
  2. 2021COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentencescited by 40
  3. 2021Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schemacited by 32
  4. 2023Exploring the Trade-Offs: Unified Large Language Models vs Local Fine-Tuned Models for Highly-Specific Radiology NLI Taskcited by 32
  5. 2021Supervising Model Attention with Human Explanations for Robust Natural Language Inferencecited by 29
  6. 2022DISCO: Distilling Counterfactuals with Large Language Modelscited by 29
  7. 2022Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learningcited by 25
  8. 2021Analyzing Dynamic Adversarial Training Data in the Limitcited by 24
  9. 2021IndoNLI: A Natural Language Inference Dataset for Indonesiancited by 24
  10. 2021Avoiding Inference Heuristics in Few-shot Prompt-based Finetuningcited by 23
  11. 2021Rissanen Data Analysis: Examining Dataset Characteristics via Description Lengthcited by 22
  12. 2024SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trialscited by 20
  13. 2023NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reportscited by 18
  14. 2021MedNLI Is Not Immune: Natural Language Inference Artifacts in the Clinical Domaincited by 17
  15. 2021Exploring Transitivity in Neural NLI Models through Veridicalitycited by 15
  16. 2021Mitigating the Position Bias of Transformer Models in Passage Re-Rankingcited by 10
  17. 2021SyGNS: A Systematic Generalization Testbed Based on Natural Language Semanticscited by 7
  18. 2022Partial-input baselines show that NLI models can ignore context, but they don'tcited by 6
  19. 2022LawngNLI: A Long-Premise Benchmark for In-Domain Generalization from Short to Long Contexts and for Implication-Based Retrievalcited by 6
  20. 2023Investigating Multi-source Active Learning for Natural Language Inferencecited by 6
  21. 2021Robustness and Sensitivity of BERT Models Predicting Alzheimer's Disease from Textcited by 5
  22. 2025Training-free LLM Merging for Multi-task Learningcited by 5
  23. 2023A Simple yet Effective Self-Debiasing Framework for Transformer Modelscited by 4
  24. 2023SpaceNLI: Evaluating the Consistency of Predicting Inferences in Spacecited by 4
  25. 2024How Hard is this Test Set? NLI Characterization by Exploiting Training Dynamicscited by 4
  26. 2021Posthoc Verification and the Fallibility of the Ground Truthcited by 3
  27. 2021A Generative Approach for Mitigating Structural Biases in Natural Language Inferencecited by 3
  28. 2023Stubborn Lexical Bias in Data and Modelscited by 3
  29. 2024XNLIeu: a dataset for cross-lingual NLI in Basquecited by 3
  30. 2022A Proposal to Study "Is High Quality Data All We Need?"cited by 2
  31. 2022Multilingual CheckList: Generation and Evaluationcited by 2
  32. 2022DECK: Behavioral Tests to Improve Interpretability and Generalizability of BERT Models Detecting Depression from Textcited by 2
  33. 2022A Survey of Parameters Associated with the Quality of Benchmarks in NLPcited by 1
  34. 2023Formal Proofs as Structured Explanations: Proposing Several Tasks on Explainable Natural Language Inferencecited by 1
  35. 2025Biases in Large Language Model-Elicited Text: A Case Study in Natural Language Inferencecited by 1
  36. 2025Evaluating the Scalability and Adversarial Generalization of GRPO-Trained NLI Modelscited by 1
  37. 2021A Logic-Based Framework for Natural Language Inference in Dutchcited by 0
  38. 2022How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasetscited by 0

50 most cited of 58 All 58 (JSON)