# Improving QA Model Performance with Cartographic Inoculation

arXiv: https://arxiv.org/abs/2401.17498 · HTML: https://mlsanity.com/paper/2401.17498/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 14 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Builds on (most cited first, 11 of 11)

- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020)
- [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018)
- [Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual](https://mlsanity.com/paper/1908.10763/) (2019)
- [Competency Problems: On Finding and Removing Artifacts in Language Data](https://mlsanity.com/paper/2104.08646/) (2021)
- [How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks](https://mlsanity.com/paper/1808.04926/) (2018)
- [Inoculation by Fine-Tuning: A Method for Analyzing Challenge Datasets](https://mlsanity.com/paper/1904.02668/) (2019)
- [Learning with Instance Bundles for Reading Comprehension](https://mlsanity.com/paper/2104.08735/) (2021)
- [Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets](https://mlsanity.com/paper/2311.08662/) (2023)

## Built on by (most cited first, 1 of 1)

- [PEDANTS: Cheap but Effective and Interpretable Answer Equivalence](https://mlsanity.com/paper/2402.11161/) (2024)

## Abstract (arXiv)

QA models are faced with complex and open-ended contextual reasoning problems, but can often learn well-performing solution heuristics by exploiting dataset-specific patterns in their training data. These patterns, or "dataset artifacts", reduce the model's ability to generalize to real-world QA problems. Utilizing an ElectraSmallDiscriminator model trained for QA, we analyze the impacts and incidence of dataset artifacts using an adversarial challenge set designed to confuse models reliant on artifacts for prediction. Extending existing work on methods for mitigating artifact impacts, we propose cartographic inoculation, a novel method that fine-tunes models on an optimized subset of the challenge data to reduce model reliance on dataset artifacts. We show that by selectively fine-tuning a model on ambiguous adversarial examples from a challenge set, significant performance improvements can be made on the full challenge dataset with minimal loss of model generalizability to other challenging environments and QA datasets.