mlsanityData to 14 Oct 2026

Practical Benefits of Feature Feedback Under Distribution Shift

arXiv 2110.07566 ↗First posted 14 Oct 2021cs.CL cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

In attempts to develop sample-efficient and interpretable algorithms, researcher have explored myriad mechanisms for collecting and exploiting feature feedback (or rationales) auxiliary annotations provided for training (but not test) instances that highlight salient evidence. Examples include bounding boxes around objects and salient spans in text. Despite its intuitive appeal, feature feedback has not delivered significant gains in practical problems as assessed on iid holdout sets. However, recent works on counterfactually augmented data suggest an alternative benefit of supplemental annotations, beyond interpretability: lessening sensitivity to spurious patterns and consequently delivering gains in out-of-domain evaluations. We speculate that while existing methods for incorporating feature feedback have delivered negligible in-sample performance gains, they may nevertheless provide out-of-domain benefits. Our experiments addressing sentiment analysis, show that feature feedback methods perform significantly better on various natural out-of-domain datasets despite comparable in-domain evaluations. By contrast, performance on natural language inference remains comparable. Finally, we compare those tasks where feature feedback does (and does not) help.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 19 of its 31 references that are in the map. First posted on arXiv 14 Oct 2021.

Builds on

19 of 31 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  3. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
  4. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
  5. 2017Right for the Right Reasons: Training Differentiable Models by Constraining their Explanationscited by 294
  6. 2016Rationalizing Neural Predictionscited by 281
  7. 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
  8. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 210
  9. 2020Learning to Faithfully Rationalize by Constructioncited by 113
  10. 2017Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisiscited by 96
  11. 2019Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claimscited by 74
  12. 2020Explaining The Efficacy of Counterfactually Augmented Datacited by 65
7 more
  1. 2019Inferring Which Medical Treatments Work from Reports of Clinical Trialscited by 64
  2. 2020QED: A Framework and Dataset for Explanations in Question Answeringcited by 55
  3. 2016Rationale-Augmented Convolutional Neural Networks for Text Classificationcited by 39
  4. 2020Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Datacited by 27
  5. 2020Weakly- and Semi-supervised Evidence Extractioncited by 16
  6. 2020Robust Learning from Discriminative Feature Feedbackcited by 7
  7. 2021Model Agnostic Answer Reranking System for Adversarial Question Answeringcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 1.

5 papers.

  1. 2024Closed-Loop Unsupervised Representation Disentanglement with β-VAE Distillation and Diffusion Probabilistic Feedbackcited by 9
  2. 2023Unsupervised domain adaptation by learning using privileged informationcited by 4
  3. 2022Learning to Ignore Adversarial Attackscited by 2
  4. 2023Zero-Shot Cross-Lingual Sentiment Classification under Distribution Shift: an Exploratory Studycited by 2
  5. 2022Robustifying Sentiment Classification by Maximally Exploiting Few Counterfactualscited by 0