# SCALE: Towards Collaborative Content Analysis in Social Science with Large Language Model Agents and Human Intervention

arXiv: https://arxiv.org/abs/2502.10937 · HTML: https://mlsanity.com/paper/2502.10937/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 12 of its 57 references that are in the map.
- First posted on arXiv 16 Feb 2025.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [Human-in-the-loop Artificial Intelligence](https://mlsanity.com/paper/1710.08191/) (2017)
- [Transitioning to human interaction with AI systems: New challenges and opportunities for HCI professionals to enable human-centered AI](https://mlsanity.com/paper/2105.05424/) (2021)
- [Ontology-Aware RAG for Improved Question-Answering in Cybersecurity Education](https://mlsanity.com/paper/2412.14191/) (2024)
- [Rise of the HaCRS: Augmenting Autonomous Cyber Reasoning Systems with Human Assistance](https://mlsanity.com/paper/1708.02749/) (2017)

## Built on by (most cited first, 7 of 7)

- [Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens](https://mlsanity.com/paper/2508.01191/) (2025)
- [ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents](https://mlsanity.com/paper/2601.12294/) (2026)
- [Window Token Concatenation for Efficient Visual Large Language Models](https://mlsanity.com/paper/2504.04024/) (2025)
- [Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification](https://mlsanity.com/paper/2507.05010/) (2025)
- [CyberBOT: Towards Reliable Cybersecurity Education via Ontology-Grounded Retrieval Augmented Generation](https://mlsanity.com/paper/2504.00389/) (2025)
- [Causality Guided Representation Learning for Cross-Style Hate Speech Detection](https://mlsanity.com/paper/2510.07707/) (2025)
- [REC-CBM: Rubric-Aware Error-Correction Concept Bottleneck Models for Trustworthy Open-Ended Grading](https://mlsanity.com/paper/2605.27402/) (2026)

## Abstract (arXiv)

Content analysis breaks down complex and unstructured texts into theory-informed numerical categories. Particularly, in social science, this process usually relies on multiple rounds of manual annotation, domain expert discussion, and rule-based refinement. In this paper, we introduce SCALE, a novel multi-agent framework that effectively underlineSimulates underlineContent underlineAnalysis via underlineLarge language model (LLM) agunderlineEnts. SCALE imitates key phases of content analysis, including text coding, collaborative discussion, and dynamic codebook evolution, capturing the reflective depth and adaptive discussions of human researchers. Furthermore, by integrating diverse modes of human intervention, SCALE is augmented with expert input to further enhance its performance. Extensive evaluations on real-world datasets demonstrate that SCALE achieves human-approximated performance across various complex content analysis tasks, offering an innovative potential for future social science research.