# Do LLMs Find Human Answers To Fact-Driven Questions Perplexing? A Case Study on Reddit

arXiv: https://arxiv.org/abs/2404.01147 · HTML: https://mlsanity.com/paper/2404.01147/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 21 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 8 of 8)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://mlsanity.com/paper/2310.07521/) (2023)
- [TWEETQA: A Social Media Focused Question Answering Dataset](https://mlsanity.com/paper/1907.06292/) (2019)
- [What do LLMs Know about Financial Markets? A Case Study on Reddit Market Sentiment Analysis](https://mlsanity.com/paper/2212.11311/) (2022)
- [Engagement Patterns of Peer-to-Peer Interactions on Mental Health Platforms](https://mlsanity.com/paper/2004.04999/) (2020)
- [Scope of Large Language Models for Mining Emerging Opinions in Online Health Discourse](https://mlsanity.com/paper/2403.03336/) (2024)

## Built on by (most cited first, 2 of 2)

- [Unveiling Entity-Level Unlearning for Large Language Models: A Comprehensive Analysis](https://mlsanity.com/paper/2406.15796/) (2024)
- [Measuring Distribution Shift in User Prompts and Its Effects on LLM Performance](https://mlsanity.com/paper/2604.17650/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have been shown to be proficient in correctly answering questions in the context of online discourse. However, the study of using LLMs to model human-like answers to fact-driven social media questions is still under-explored. In this work, we investigate how LLMs model the wide variety of human answers to fact-driven questions posed on several topic-specific Reddit communities, or subreddits. We collect and release a dataset of 409 fact-driven questions and 7,534 diverse, human-rated answers from 15 r/Ask{Topic} communities across 3 categories: profession, social identity, and geographic location. We find that LLMs are considerably better at modeling highly-rated human answers to such questions, as opposed to poorly-rated human answers. We present several directions for future research based on our initial findings.