# Natural Language Inference with Mixed Effects

arXiv: https://arxiv.org/abs/2010.10501 · HTML: https://mlsanity.com/paper/2010.10501/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2020.

## Built on by (most cited first, 9 of 9)

- [LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts](https://mlsanity.com/paper/2501.00274/) (2024)
- [Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning](https://mlsanity.com/paper/2104.08676/) (2021)
- [Compositional Evaluation on Japanese Textual Entailment and Similarity](https://mlsanity.com/paper/2208.04826/) (2022)
- [Decomposing and Recomposing Event Structure](https://mlsanity.com/paper/2103.10387/) (2021)
- [No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inference](https://mlsanity.com/paper/2306.09918/) (2023)
- [Towards More Robust Natural Language Understanding](https://mlsanity.com/paper/2112.02992/) (2021)
- [Reanalyzing L2 Preposition Learning with Bayesian Mixed Effects and a Pretrained Language Model](https://mlsanity.com/paper/2302.08150/) (2023)
- [How Grounded is Wikipedia? A Study on Structured Evidential Support and Retrieval](https://mlsanity.com/paper/2506.12637/) (2025)
- [On the Role of Citations in Preference Data](https://mlsanity.com/paper/2608.21376/) (2026)

## Abstract (arXiv)

There is growing evidence that the prevalence of disagreement in the raw annotations used to construct natural language inference datasets makes the common practice of aggregating those annotations to a single label problematic. We propose a generic method that allows one to skip the aggregation step and train on the raw annotations directly without subjecting the model to unwanted noise that can arise from annotator response biases. We demonstrate that this method, which generalizes the notion of a textitmixed effects model by incorporating textitannotator random effects into any existing neural model, improves performance over models that do not incorporate such effects.