mlsanityData to 14 Oct 2026

Rethinking STS and NLI in Large Language Models

arXiv 2309.08969 ↗First posted 16 Sep 2023cs.CLcited by 2

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Recent years have seen the rise of large language models (LLMs), where practitioners use task-specific prompts; this was shown to be effective for a variety of tasks. However, when applied to semantic textual similarity (STS) and natural language inference (NLI), the effectiveness of LLMs turns out to be limited by low-resource domain accuracy, model overconfidence, and difficulty to capture the disagreements between human judgements. With this in mind, here we try to rethink STS and NLI in the era of LLMs. We first evaluate the performance of STS and NLI in the clinical/biomedical domain, and then we assess LLMs' predictive confidence and their capability of capturing collective human opinions. We find that these old problems are still to be properly addressed in the era of LLMs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 23 of its 32 references that are in the map. First posted on arXiv 16 Sep 2023.

Builds on

23 of 32 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks1 Mar

Other references · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  6. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  7. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  9. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  11. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  12. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
10 more
  1. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  2. 2023How Language Model Hallucinations Can Snowballcited by 311
  3. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 293
  4. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  5. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 153
  6. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 151
  7. 2018MedSTS: A Resource for Clinical Semantic Textual Similaritycited by 39
  8. 2023Exploring the Trade-Offs: Unified Large Language Models vs Local Fine-Tuned Models for Highly-Specific Radiology NLI Taskcited by 32
  9. 2020UNITE: Uncertainty-based Health Risk Prediction Leveraging Multi-sourced Datacited by 9
  10. 2023Collective Human Opinions in Semantic Textual Similaritycited by 8

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1.

2 papers.

  1. 2023Factuality Challenges in the Era of Large Language Modelscited by 126
  2. 2024Lessons from the Use of Natural Language Inference (NLI) in Requirements Engineering Taskscited by 12

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting NLI: Towards Cost-Effective and Human-Aligned Metrics for Evaluating LLMs in Question Answering
  2. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition
  3. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks
  4. 2023Aligning Large Language Models with Human: A Survey
  5. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers
  6. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
  7. 2025Modeling Understanding of Story-Based Analogies Using Large Language Models
  8. 2026LLMs Can Better Capture Human Judgments--With the Right Prompts