# QuaLLM: An LLM-based Framework to Extract Quantitative Insights from Online Forums

arXiv: https://arxiv.org/abs/2405.05345 · HTML: https://mlsanity.com/paper/2405.05345/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 73 references that are in the map.
- First posted on arXiv 8 May 2024.

## Builds on (most cited first, 13 of 13)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting](https://mlsanity.com/paper/2306.17563/) (2023)
- [Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding](https://mlsanity.com/paper/2304.10548/) (2023)
- [LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis](https://mlsanity.com/paper/2310.15100/) (2023)
- [LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding](https://mlsanity.com/paper/2306.14924/) (2023)
- [From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting](https://mlsanity.com/paper/2309.04269/) (2023)
- [Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies](https://mlsanity.com/paper/2309.13063/) (2023)
- [The illusion of artificial inclusion](https://mlsanity.com/paper/2401.08572/) (2024)
- [Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Tasks](https://mlsanity.com/paper/2401.15170/) (2024)
- [No Echo in the Chambers of Political Interactions on Reddit](https://mlsanity.com/paper/2102.05477/) (2021)
- [Upvotes? Downvotes? No Votes? Understanding the relationship between reaction mechanisms and political discourse on Reddit](https://mlsanity.com/paper/2302.09540/) (2023)

## Built on by (most cited first, 8 of 8)

- [Examining the Prevalence and Dynamics of AI-Generated Media in Art Subreddits](https://mlsanity.com/paper/2410.07302/) (2024)
- [LOGOS: LLM-driven End-to-End Grounded Theory Development and Schema Induction for Qualitative Research](https://mlsanity.com/paper/2509.24294/) (2025)
- [Who's Asking? Simulating Role-Based Questions for Conversational AI Evaluation](https://mlsanity.com/paper/2510.16829/) (2025)
- [FareShare: A Tool for Labor Organizers to Estimate Lost Wages and Contest Arbitrary AI and Algorithmic Deactivations](https://mlsanity.com/paper/2505.08904/) (2025)
- [Bridging Human Interpretation and Machine Representation: A Landscape of Qualitative Data Analysis in the LLM Era](https://mlsanity.com/paper/2601.11739/) (2026)
- [Can LLMs Understand the Impact of Trauma? Costs and Benefits of LLMs Coding the Interviews of Firearm Violence Survivors](https://mlsanity.com/paper/2604.16132/) (2026)
- [What Do People Actually Want From AI? Mapping Preference Plurality](https://mlsanity.com/paper/2606.06674/) (2026)
- [How AI Coders Discuss, Disagree, and Reach Consensus: Challenges and Opportunities for LLM-Based Qualitative Coding](https://mlsanity.com/paper/2609.11109/) (2026)

## Abstract (arXiv)

Online discussion forums provide crucial data to understand the concerns of a wide range of real-world communities. However, the typical qualitative and quantitative methodologies used to analyze those data, such as thematic analysis and topic modeling, are infeasible to scale or require significant human effort to translate outputs to human readable forms. This study introduces QuaLLM, a novel LLM-based framework to analyze and extract quantitative insights from text data on online forums. The framework consists of a novel prompting and human evaluation methodology. We applied this framework to analyze over one million comments from two of Reddit's rideshare worker communities, marking the largest study of its type. We uncover significant worker concerns regarding AI and algorithmic platform decisions, responding to regulatory calls about worker insights. In short, our work sets a new precedent for AI-assisted quantitative data analysis to surface concerns from online forums.