# ExpBERT: Representation Engineering with Natural Language Explanations

arXiv: https://arxiv.org/abs/2005.01932 · HTML: https://mlsanity.com/paper/2005.01932/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 May 2020.

## Built on by (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Survey of Knowledge Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2211.05994/) (2022)
- [When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data](https://mlsanity.com/paper/2102.02201/) (2021)
- [AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning](https://mlsanity.com/paper/2102.01386/) (2021)
- [Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity](https://mlsanity.com/paper/2111.08366/) (2021)
- [Fixing Model Bugs with Natural Language Patches](https://mlsanity.com/paper/2211.03318/) (2022)
- [Large Language Model Instruction Following: A Survey of Progresses and Challenges](https://mlsanity.com/paper/2303.10475/) (2023)
- [Supervising Model Attention with Human Explanations for Robust Natural Language Inference](https://mlsanity.com/paper/2104.08142/) (2021)
- [Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deployment](https://mlsanity.com/paper/2204.03025/) (2022)
- [Understanding and Enhancing Robustness of Concept-based Models](https://mlsanity.com/paper/2211.16080/) (2022)
- [A Review of Text Style Transfer using Deep Learning](https://mlsanity.com/paper/2109.15144/) (2021)
- [A survey on improving NLP models with human explanations](https://mlsanity.com/paper/2204.08892/) (2022)
- [SsciBERT: A Pre-trained Language Model for Social Science Texts](https://mlsanity.com/paper/2206.04510/) (2022)
- [Using Natural Language Explanations to Rescale Human Judgments](https://mlsanity.com/paper/2305.14770/) (2023)
- [CLUES: A Benchmark for Learning Classifiers using Natural Language Explanations](https://mlsanity.com/paper/2204.07142/) (2022)
- [Automatic Concept Extraction for Concept Bottleneck-based Video Classification](https://mlsanity.com/paper/2206.10129/) (2022)
- [A Self-explaining Neural Architecture for Generalizable Concept Learning](https://mlsanity.com/paper/2405.00349/) (2024)
- [Enhancing Natural Language Representation with Large-Scale Out-of-Domain Commonsense](https://mlsanity.com/paper/2109.02572/) (2021)
- [SemSup: Semantic Supervision for Simple and Scalable Zero-shot Generalization](https://mlsanity.com/paper/2202.13100/) (2022)
- [Leveraging Multiple Teachers for Test-Time Adaptation of Language-Guided Classifiers](https://mlsanity.com/paper/2311.07538/) (2023)
- [Okay, Let's Do This! Modeling Event Coreference with Generated Rationales and Knowledge Distillation](https://mlsanity.com/paper/2404.03196/) (2024)
- [Improving Attributed Long-form Question Answering with Intent Awareness](https://mlsanity.com/paper/2603.27435/) (2026)
- [Open Rule Induction](https://mlsanity.com/paper/2110.13577/) (2021)
- [Teach me how to Label: Labeling Functions from Natural Language with Text-to-text Transformers](https://mlsanity.com/paper/2101.07138/) (2021)
- [Bootstrapping Relation Extractors using Syntactic Search by Examples](https://mlsanity.com/paper/2102.05007/) (2021)
- [Technical Language Supervision for Intelligent Fault Diagnosis in Process Industry](https://mlsanity.com/paper/2112.07356/) (2021)
- [Exploring Automatically Perturbed Natural Language Explanations in Relation Extraction](https://mlsanity.com/paper/2305.15520/) (2023)
- [Reading Is Believing: Revisiting Language Bottleneck Models for Image Classification](https://mlsanity.com/paper/2406.15816/) (2024)

## Abstract (arXiv)

Suppose we want to specify the inductive bias that married couples typically go on honeymoons for the task of extracting pairs of spouses from text. In this paper, we allow model developers to specify these types of inductive biases as natural language explanations. We use BERT fine-tuned on MultiNLI to ``interpret'' these explanations with respect to the input sentence, producing explanation-guided representations of the input. Across three relation extraction tasks, our method, ExpBERT, matches a BERT baseline but with 3--20x less labeled data and improves on the baseline by 3--10 F1 points with the same amount of labeled data.