# Uncertain Natural Language Inference

arXiv: https://arxiv.org/abs/1909.03042 · HTML: https://mlsanity.com/paper/1909.03042/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Sep 2019.

## Built on by (most cited first, 30 of 35)

- [We're Afraid Language Models Aren't Modeling Ambiguity](https://mlsanity.com/paper/2304.14399/) (2023)
- [Stop Measuring Calibration When Humans Disagree](https://mlsanity.com/paper/2210.16133/) (2022)
- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [Natural Language Deduction through Search over Statement Compositions](https://mlsanity.com/paper/2201.06028/) (2022)
- [Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning](https://mlsanity.com/paper/2104.08676/) (2021)
- [Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge](https://mlsanity.com/paper/2410.03775/) (2024)
- [Learning with Different Amounts of Annotation: From Zero to Many Labels](https://mlsanity.com/paper/2109.04408/) (2021)
- [MPCHAT: Towards Multimodal Persona-Grounded Conversation](https://mlsanity.com/paper/2305.17388/) (2023)
- [Embracing Ambiguity: Shifting the Training Target of NLI Models](https://mlsanity.com/paper/2106.03020/) (2021)
- [NOPE: A Corpus of Naturally-Occurring Presuppositions in English](https://mlsanity.com/paper/2109.06987/) (2021)
- [Refined Commonsense Knowledge from Large-Scale Web Contents](https://mlsanity.com/paper/2112.04596/) (2021)
- [BIRD: A Trustworthy Bayesian Inference Framework for Large Language Models](https://mlsanity.com/paper/2404.12494/) (2024)
- [Zero and Few-shot Semantic Parsing with Ambiguous Inputs](https://mlsanity.com/paper/2306.00824/) (2023)
- [Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design](https://mlsanity.com/paper/2304.00815/) (2023)
- [Ambiguous Images With Human Judgments for Robust Visual Event Classification](https://mlsanity.com/paper/2210.03102/) (2022)
- [Using Natural Language Explanations to Rescale Human Judgments](https://mlsanity.com/paper/2305.14770/) (2023)
- [Clean or Annotate: How to Spend a Limited Data Collection Budget](https://mlsanity.com/paper/2110.08355/) (2021)
- [MegaWika: Millions of reports and their sources across 50 diverse languages](https://mlsanity.com/paper/2307.07049/) (2023)
- [Always Tell Me The Odds: Fine-grained Conditional Probability Estimation](https://mlsanity.com/paper/2505.01595/) (2025)
- [Collective Human Opinions in Semantic Textual Similarity](https://mlsanity.com/paper/2308.04114/) (2023)
- [Probing neural language models for understanding of words of estimative probability](https://mlsanity.com/paper/2211.03358/) (2022)
- [Capturing Label Distribution: A Case Study in NLI](https://mlsanity.com/paper/2102.06859/) (2021)
- [Investigating Multi-source Active Learning for Natural Language Inference](https://mlsanity.com/paper/2302.06976/) (2023)
- [Uncertainty-Aware Natural Language Inference with Stochastic Weight Averaging](https://mlsanity.com/paper/2304.04726/) (2023)
- [No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inference](https://mlsanity.com/paper/2306.09918/) (2023)
- [Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios](https://mlsanity.com/paper/2209.07760/) (2022)
- [Pneg: Prompt-based Negative Response Generation for Dialogue Response Selection Task](https://mlsanity.com/paper/2210.17238/) (2022)
- [Defeasible Visual Entailment: Benchmark, Evaluator, and Reward-Driven Optimization](https://mlsanity.com/paper/2412.16232/) (2024)
- [Interpretable Multimodal Out-of-context Detection with Soft Logic Regularization](https://mlsanity.com/paper/2406.04756/) (2024)
- [Process Supervision of Confidence Margin for Calibrated LLM Reasoning](https://mlsanity.com/paper/2604.23333/) (2026)

## Abstract (arXiv)

We introduce Uncertain Natural Language Inference (UNLI), a refinement of Natural Language Inference (NLI) that shifts away from categorical labels, targeting instead the direct prediction of subjective probability assessments. We demonstrate the feasibility of collecting annotations for UNLI by relabeling a portion of the SNLI dataset under a probabilistic scale, where items even with the same categorical label differ in how likely people judge them to be true given a premise. We describe a direct scalar regression modeling approach, and find that existing categorically labeled NLI data can be used in pre-training. Our best models approach human performance, demonstrating models may be capable of more subtle inferences than the categorical bin assignment employed in current NLI tasks.