# Recalibrating the Compass: Integrating Large Language Models into Classical Research Methods

arXiv: https://arxiv.org/abs/2505.19402 · HTML: https://mlsanity.com/paper/2505.19402/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 82 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 30 of 31)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives](https://mlsanity.com/paper/2312.11970/) (2023)
- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [Do LLMs exhibit human-like response biases? A case study in survey design](https://mlsanity.com/paper/2311.04076/) (2023)
- [Simulating Opinion Dynamics with Networks of LLM-based Agents](https://mlsanity.com/paper/2311.09618/) (2023)
- [OASIS: Open Agent Social Interaction Simulations with One Million Agents](https://mlsanity.com/paper/2411.11581/) (2024)
- [Aligning Language Models to User Opinions](https://mlsanity.com/paper/2305.14929/) (2023)
- [From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents](https://mlsanity.com/paper/2412.03563/) (2024)
- [The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs](https://mlsanity.com/paper/2501.10970/) (2025)
- ["HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social media](https://mlsanity.com/paper/2304.10619/) (2023)
- [Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy](https://mlsanity.com/paper/2407.06813/) (2024)
- [Language Models Trained on Media Diets Can Predict Public Opinion](https://mlsanity.com/paper/2303.16779/) (2023)
- [Human-in-the-loop or AI-in-the-loop? Automate or Collaborate?](https://mlsanity.com/paper/2412.14232/) (2024)
- [Fact-checking information from large language models can decrease headline discernment](https://mlsanity.com/paper/2308.10800/) (2023)
- [Ask LLMs Directly, "What shapes your bias?": Measuring Social Bias in Large Language Models](https://mlsanity.com/paper/2406.04064/) (2024)
- [Collaborating with AI Agents: Field Experiments on Teamwork, Productivity, and Performance](https://mlsanity.com/paper/2503.18238/) (2025)
- [Modelling Political Coalition Negotiations Using LLM-based Agents](https://mlsanity.com/paper/2402.11712/) (2024)
- [Are Large Language Models (LLMs) Good Social Predictors?](https://mlsanity.com/paper/2402.12620/) (2024)
- [Beyond Partisan Leaning: A Comparative Analysis of Political Bias in Large Language Models](https://mlsanity.com/paper/2412.16746/) (2024)
- [Can Generative AI agents behave like humans? Evidence from laboratory market experiments](https://mlsanity.com/paper/2505.07457/) (2025)
- [Engagement-Driven Content Generation with Large Language Models](https://mlsanity.com/paper/2411.13187/) (2024)
- [Synthesizing Public Opinions with LLMs: Role Creation, Impacts, and the Future to eDemorcacy](https://mlsanity.com/paper/2504.00241/) (2025)
- [A large-scale evaluation of commonsense knowledge in humans and large language models](https://mlsanity.com/paper/2505.10309/) (2025)

## Abstract (arXiv)

This paper examines how large language models (LLMs) are transforming core quantitative methods in communication research in particular, and in the social sciences more broadly-namely, content analysis, survey research, and experimental studies. Rather than replacing classical approaches, LLMs introduce new possibilities for coding and interpreting text, simulating dynamic respondents, and generating personalized and interactive stimuli. Drawing on recent interdisciplinary work, the paper highlights both the potential and limitations of LLMs as research tools, including issues of validity, bias, and interpretability. To situate these developments theoretically, the paper revisits Lasswell's foundational framework -- "Who says what, in which channel, to whom, with what effect?" -- and demonstrates how LLMs reconfigure message studies, audience analysis, and effects research by enabling interpretive variation, audience trajectory modeling, and counterfactual experimentation. Revisiting the metaphor of the methodological compass, the paper argues that classical research logics remain essential as the field integrates LLMs and generative AI. By treating LLMs not only as technical instruments but also as epistemic and cultural tools, the paper calls for thoughtful, rigorous, and imaginative use of LLMs in future communication and social science research.