# Potential and Perils of Large Language Models as Judges of Unstructured Textual Data

arXiv: https://arxiv.org/abs/2501.08167 · HTML: https://mlsanity.com/paper/2501.08167/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jan 2025.

## Built on by (most cited first, 11 of 11)

- [DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process](https://mlsanity.com/paper/2503.08569/) (2025)
- [Evaluating LLM Agent Collusion in Double Auctions](https://mlsanity.com/paper/2507.01413/) (2025)
- [IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding](https://mlsanity.com/paper/2501.15747/) (2025)
- [AutoSurvey2: Empowering Researchers with Next Level Automated Literature Surveys](https://mlsanity.com/paper/2510.26012/) (2025)
- [NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment](https://mlsanity.com/paper/2604.11543/) (2026)
- [AccessEval: Benchmarking Disability Bias in Large Language Models](https://mlsanity.com/paper/2509.22703/) (2025)
- [Position: Thematic Analysis of Unstructured Clinical Transcripts with Large Language Models](https://mlsanity.com/paper/2509.14597/) (2025)
- [Through the Judge's Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters](https://mlsanity.com/paper/2510.25860/) (2025)
- [Transforming Expert Knowledge into Scalable Ontology via Large Language Models](https://mlsanity.com/paper/2506.08422/) (2025)
- [Simple Role Assignment is Extraordinarily Effective for Safety Alignment](https://mlsanity.com/paper/2602.00061/) (2026)
- [Simulating Meaning, Nevermore! Introducing ICR: A Semiotic-Hermeneutic Metric for Evaluating Meaning in LLM Text Summaries](https://mlsanity.com/paper/2603.04413/) (2026)

## Abstract (arXiv)

Rapid advancements in large language models have unlocked remarkable capabilities when it comes to processing and summarizing unstructured text data. This has implications for the analysis of rich, open-ended datasets, such as survey responses, where LLMs hold the promise of efficiently distilling key themes and sentiments. However, as organizations increasingly turn to these powerful AI systems to make sense of textual feedback, a critical question arises, can we trust LLMs to accurately represent the perspectives contained within these text based datasets? While LLMs excel at generating human-like summaries, there is a risk that their outputs may inadvertently diverge from the true substance of the original responses. Discrepancies between the LLM-generated outputs and the actual themes present in the data could lead to flawed decision-making, with far-reaching consequences for organizations. This research investigates the effectiveness of LLM-as-judge models to evaluate the thematic alignment of summaries generated by other LLMs. We utilized an Anthropic Claude model to generate thematic summaries from open-ended survey responses, with Amazon's Titan Express, Nova Pro, and Meta's Llama serving as judges. This LLM-as-judge approach was compared to human evaluations using Cohen's kappa, Spearman's rho, and Krippendorff's alpha, validating a scalable alternative to traditional human centric evaluation methods. Our findings reveal that while LLM-as-judge offer a scalable solution comparable to human raters, humans may still excel at detecting subtle, context-specific nuances. Our research contributes to the growing body of knowledge on AI assisted text analysis. Further, we provide recommendations for future research, emphasizing the need for careful consideration when generalizing LLM-as-judge models across various contexts and use cases.