mlsanityData to 14 Oct 2026

MRCLens: an MRC Dataset Bias Detection Toolkit

arXiv 2207.08943 ↗First posted 18 Jul 2022cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Many recent neural models have shown remarkable empirical results in Machine Reading Comprehension, but evidence suggests sometimes the models take advantage of dataset biases to predict and fail to generalize on out-of-sample data. While many other approaches have been proposed to address this issue from the computation perspective such as new architectures or training procedures, we believe a method that allows researchers to discover biases, and adjust the data or the models in an earlier stage will be beneficial. Thus, we introduce MRCLens, a toolkit that detects whether biases exist before users train the full model. For the convenience of introducing the toolkit, we also provide a categorization of common biases in MRC.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 18 Jul 2022.

Builds on

30 of 43 references.

References · 30

  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  3. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  4. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  5. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  6. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  7. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
  8. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
  9. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 280
  10. 2016Bidirectional Attention Flow for Machine Comprehensioncited by 278
  11. 2018Audio Adversarial Examples: Targeted Attacks on Speech-to-Textcited by 274
  12. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 214
18 more
  1. 2016Enhanced LSTM for Natural Language Inferencecited by 200
  2. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 184
  3. 2018QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehensioncited by 164
  4. 2018Breaking NLI Systems with Sentences that Require Simple Lexical Inferencescited by 134
  5. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  6. 2018How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarkscited by 88
  7. 2019Learning and Evaluating General Linguistic Intelligencecited by 79
  8. 2018Did the Model Understand the Question?cited by 60
  9. 2018What Makes Reading Comprehension Questions Easier?cited by 60
  10. 2019MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehensioncited by 60
  11. 2019Understanding Dataset Design Choices for Multi-hop Reasoningcited by 50
  12. 2019Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasetscited by 46
  13. 2020What do Models Learn from Question Answering Datasets?cited by 45
  14. 2017Making Neural QA as Simple as Possible but not Simplercited by 33
  15. 2018Efficient and Robust Question Answering from Minimal Context over Documentscited by 26
  16. 2020Probabilistic Assumptions Matter: Improved Models for Distantly-Supervised Document-Level Question Answeringcited by 17
  17. 2018What If We Simply Swap the Two Text Fragments? A Straightforward yet Effective Way to Test the Robustness of Methods to Confounding Signals in Nature Language Inference Taskscited by 14
  18. 2018QADiver: Interactive Framework for Diagnosing QA Modelscited by 3

Built on by

Not cited yet.