# From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviews

arXiv: https://arxiv.org/abs/2312.01202 · HTML: https://mlsanity.com/paper/2312.01202/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 82 references that are in the map.
- First posted on arXiv 2 Dec 2023.

## Builds on (most cited first, 19 of 19)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [Model-Based Deep Learning](https://mlsanity.com/paper/2012.08405/) (2020)
- [The Science of Detecting LLM-Generated Texts](https://mlsanity.com/paper/2303.07205/) (2023)
- [Is ChatGPT a Good Sentiment Analyzer? A Preliminary Study](https://mlsanity.com/paper/2304.04339/) (2023)
- [Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding](https://mlsanity.com/paper/2304.10548/) (2023)
- [Is GPT-4 a Good Data Analyst?](https://mlsanity.com/paper/2305.15038/) (2023)
- [LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis](https://mlsanity.com/paper/2310.15100/) (2023)
- [LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding](https://mlsanity.com/paper/2306.14924/) (2023)
- [Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potential](https://mlsanity.com/paper/2303.09038/) (2023)
- [SentimentGPT: Exploiting GPT for Advanced Sentiment Analysis and its Departure from Current Machine Learning](https://mlsanity.com/paper/2307.10234/) (2023)
- [Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Texts](https://mlsanity.com/paper/2305.04417/) (2023)
- [Using Large Language Models for Qualitative Analysis can Introduce Serious Bias](https://mlsanity.com/paper/2309.17147/) (2023)
- [Deep Sentiment Classification and Topic Discovery on Novel Coronavirus or COVID-19 Online Discussions: NLP Using LSTM Recurrent Neural Network Approach](https://mlsanity.com/paper/2004.11695/) (2020)
- [Leveraging ChatGPT As Text Annotation Tool For Sentiment Analysis](https://mlsanity.com/paper/2306.17177/) (2023)
- [Large Language Models for Propaganda Detection](https://mlsanity.com/paper/2310.06422/) (2023)

## Built on by (most cited first, 9 of 9)

- [AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot](https://mlsanity.com/paper/2604.13940/) (2026)
- [Assessing and Verifying Task Utility in LLM-Powered Applications](https://mlsanity.com/paper/2405.02178/) (2024)
- [Towards better Human-Agent Alignment: Assessing Task Utility in LLM-Powered Applications](https://mlsanity.com/paper/2402.09015/) (2024)
- [Large Language Models in Politics and Democracy: A Comprehensive Survey](https://mlsanity.com/paper/2412.04498/) (2024)
- [A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards](https://mlsanity.com/paper/2505.13766/) (2025)
- [How K-12 Educators Use AI: LLM-Assisted Qualitative Analysis at Scale](https://mlsanity.com/paper/2507.17985/) (2025)
- [ParliaBench: An Evaluation and Benchmarking Framework for LLM-Generated Parliamentary Speech](https://mlsanity.com/paper/2511.08247/) (2025)
- [How Trustworthy Are LLM-as-Judge Ratings for Interpretive Responses? Implications for Qualitative Research Workflows](https://mlsanity.com/paper/2604.00008/) (2026)
- [Large Language Models, Machine-Assisted Cultural Analytics, and the Agentic Turn](https://mlsanity.com/paper/2610.12792/) (2026)

## Abstract (arXiv)

Obtaining stakeholders' diverse experiences and opinions about current policy in a timely manner is crucial for policymakers to identify strengths and gaps in resource allocation, thereby supporting effective policy design and implementation. However, manually coding even moderately sized interview texts or open-ended survey responses from stakeholders can often be labor-intensive and time-consuming. This study explores the integration of Large Language Models (LLMs)--like GPT-4--with human expertise to enhance text analysis of stakeholder interviews regarding K-12 education policy within one U.S. state. Employing a mixed-methods approach, human experts developed a codebook and coding processes as informed by domain knowledge and unsupervised topic modeling results. They then designed prompts to guide GPT-4 analysis and iteratively evaluate different prompts' performances. This combined human-computer method enabled nuanced thematic and sentiment analysis. Results reveal that while GPT-4 thematic coding aligned with human coding by 77.89% at specific themes, expanding to broader themes increased congruence to 96.02%, surpassing traditional Natural Language Processing (NLP) methods by over 25%. Additionally, GPT-4 is more closely matched to expert sentiment analysis than lexicon-based methods. Findings from quantitative measures and qualitative reviews underscore the complementary roles of human domain expertise and automated analysis as LLMs offer new perspectives and coding consistency. The human-computer interactive approach enhances efficiency, validity, and interpretability of educational policy research.