mlsanityData to 14 Oct 2026

Recalibrating the Compass: Integrating Large Language Models into Classical Research Methods

arXiv 2505.19402 ↗First posted 26 May 2025cs.AI cs.CYcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper examines how large language models (LLMs) are transforming core quantitative methods in communication research in particular, and in the social sciences more broadly-namely, content analysis, survey research, and experimental studies. Rather than replacing classical approaches, LLMs introduce new possibilities for coding and interpreting text, simulating dynamic respondents, and generating personalized and interactive stimuli. Drawing on recent interdisciplinary work, the paper highlights both the potential and limitations of LLMs as research tools, including issues of validity, bias, and interpretability. To situate these developments theoretically, the paper revisits Lasswell's foundational framework -- "Who says what, in which channel, to whom, with what effect?" -- and demonstrates how LLMs reconfigure message studies, audience analysis, and effects research by enabling interpretive variation, audience trajectory modeling, and counterfactual experimentation. Revisiting the metaphor of the methodological compass, the paper argues that classical research logics remain essential as the field integrates LLMs and generative AI. By treating LLMs not only as technical instruments but also as epistemic and cultural tools, the paper calls for thoughtful, rigorous, and imaginative use of LLMs in future communication and social science research.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 82 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

31 of 82 references.

References · 31

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  3. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  4. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  6. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  7. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  8. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  9. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
  10. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 246
  11. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 180
  12. 2023Simulating Opinion Dynamics with Networks of LLM-based Agentscited by 154
19 more
  1. 2024OASIS: Open Agent Social Interaction Simulations with One Million Agentscited by 132
  2. 2023Aligning Language Models to User Opinionscited by 99
  3. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 82
  4. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 74
  5. 2023"HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social mediacited by 45
  6. 2024Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacycited by 36
  7. 2023Language Models Trained on Media Diets Can Predict Public Opinioncited by 30
  8. 2024Human-in-the-loop or AI-in-the-loop? Automate or Collaborate?cited by 23
  9. 2023Fact-checking information from large language models can decrease headline discernmentcited by 22
  10. 2024Ask LLMs Directly, "What shapes your bias?": Measuring Social Bias in Large Language Modelscited by 14
  11. 2025Collaborating with AI Agents: Field Experiments on Teamwork, Productivity, and Performancecited by 14
  12. 2024Modelling Political Coalition Negotiations Using LLM-based Agentscited by 13
  13. 2024Are Large Language Models (LLMs) Good Social Predictors?cited by 13
  14. 2024Beyond Partisan Leaning: A Comparative Analysis of Political Bias in Large Language Modelscited by 13
  15. 2025Can Generative AI agents behave like humans? Evidence from laboratory market experimentscited by 10
  16. 2024Engagement-Driven Content Generation with Large Language Modelscited by 5
  17. 2025Synthesizing Public Opinions with LLMs: Role Creation, Impacts, and the Future to eDemorcacycited by 5
  18. 2025A large-scale evaluation of commonsense knowledge in humans and large language modelscited by 4
  19. 2025Predicting Field Experiments with Large Language Modelscited by 2

Built on by

Not cited yet.