# Can Large Language Models Discern Evidence for Scientific Hypotheses? Case Studies in the Social Sciences

arXiv: https://arxiv.org/abs/2309.06578 · HTML: https://mlsanity.com/paper/2309.06578/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 7 Sep 2023.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [Making Large Language Models Better Reasoners with Step-Aware Verifier](https://mlsanity.com/paper/2206.02336/) (2022)
- [Explainable Automated Fact-Checking for Public Health Claims](https://mlsanity.com/paper/2010.09926/) (2020)
- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims](https://mlsanity.com/paper/2012.00614/) (2020)
- [Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction](https://mlsanity.com/paper/2209.11515/) (2022)
- [COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic](https://mlsanity.com/paper/2106.03794/) (2021)
- [SciFact-Open: Towards open-domain scientific claim verification](https://mlsanity.com/paper/2210.13777/) (2022)
- [UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation](https://mlsanity.com/paper/2303.08518/) (2023)
- [ChatGPT as a Factual Inconsistency Evaluator for Text Summarization](https://mlsanity.com/paper/2303.15621/) (2023)
- [A Survey on Natural Language Processing for Fake News Detection](https://mlsanity.com/paper/1811.00770/) (2018)
- [Scientific Fact-Checking: A Survey of Resources and Approaches](https://mlsanity.com/paper/2305.16859/) (2023)
- [Scientific Claim Verification with VERT5ERINI](https://mlsanity.com/paper/2010.11930/) (2020)
- [Pushing the Limits of ChatGPT on NLP Tasks](https://mlsanity.com/paper/2306.09719/) (2023)
- [Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds](https://mlsanity.com/paper/2305.14785/) (2023)

## Built on by (most cited first, 6 of 6)

- [Using Large Language Models for Automated Grading of Student Writing about Science](https://mlsanity.com/paper/2412.18719/) (2024)
- [SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence](https://mlsanity.com/paper/2512.22334/) (2025)
- [Assessing the Effectiveness of GPT-4o in Climate Change Evidence Synthesis and Systematic Assessments: Preliminary Insights](https://mlsanity.com/paper/2407.12826/) (2024)
- [Speculative Exploration on the Concept of Artificial Agents Conducting Autonomous Research](https://mlsanity.com/paper/2312.03497/) (2023)
- [What Are Research Hypotheses?](https://mlsanity.com/paper/2509.00185/) (2025)
- [Detection and Interpretability Analysis of Quotation Errors by Large Language Models](https://mlsanity.com/paper/2606.08589/) (2026)

## Abstract (arXiv)

Hypothesis formulation and testing are central to empirical research. A strong hypothesis is a best guess based on existing evidence and informed by a comprehensive view of relevant literature. However, with exponential increase in the number of scientific articles published annually, manual aggregation and synthesis of evidence related to a given hypothesis is a challenge. Our work explores the ability of current large language models (LLMs) to discern evidence in support or refute of specific hypotheses based on the text of scientific abstracts. We share a novel dataset for the task of scientific hypothesis evidencing using community-driven annotations of studies in the social sciences. We compare the performance of LLMs to several state-of-the-art benchmarks and highlight opportunities for future research in this area. The dataset is available at https://github.com/Sai90000/ScientificHypothesisEvidencing.git