mlsanityData to 7 Oct 2026

Can Large Language Models Transform Computational Social Science?

arXiv 2305.03514 ↗First posted 12 Apr 2023cs.CL cs.LGcited by 245

Not on the map. Why?

Cited by 39 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) are capable of successfully performing many language processing tasks zero-shot (without training data). If zero-shot LLMs can also reliably classify and explain social phenomena like persuasiveness and political ideology, then LLMs could augment the Computational Social Science (CSS) pipeline in important ways. This work provides a road map for using LLMs as CSS tools. Towards this end, we contribute a set of prompting best practices and an extensive evaluation pipeline to measure the zero-shot performance of 13 language models on 25 representative English CSS benchmarks. On taxonomic labeling tasks (classification), LLMs fail to outperform the best fine-tuned models but still achieve fair levels of agreement with humans. On free-form coding tasks (generation), LLMs produce explanations that often exceed the quality of crowdworkers' gold references. We conclude that the performance of today's LLMs can augment the CSS research pipeline in two ways: (1) serving as zero-shot data annotators on human annotation teams, and (2) bootstrapping challenging creative generation tasks (e.g., explaining the underlying attributes of a text). In summary, LLMs are posed to meaningfully participate in social science analysis in partnership with humans.

Key facts. Cited by 245 papers in the map (data through 7 Oct 2026). Builds on 118 of its 309 references that are in the map. First posted on arXiv 12 Apr 2023.

Builds on

118 of 309 references.

References · 118

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
38 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  8. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  9. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  10. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  13. 2023Whose Opinions Do Language Models Reflect?cited by 759
  14. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  15. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  16. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  17. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  18. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  19. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  20. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  21. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 370
  22. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  23. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  24. 2021True Few-Shot Learning with Language Modelscited by 343
  25. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  26. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  27. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 300
  28. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 251
  29. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  30. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  31. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  32. 2016Diachronic Word Embeddings Reveal Statistical Laws of Semantic Changecited by 219
  33. 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 216
  34. 2019Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Goodcited by 207
  35. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  36. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  37. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  38. 2020A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Supportcited by 182

50 most cited of 118 All 118 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 54; 2024: 71; 2025: 64; 2026: 56.

245 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  4. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  5. 2024A Roadmap to Pluralistic Alignmentcited by 175
  6. 2023MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problemscited by 128
  7. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 123
  8. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 118
  9. 2025ELEPHANT: Measuring and understanding social sycophancy in LLMscited by 115
  10. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 102
  11. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  12. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
38 more
  1. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  2. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 90
  3. 2023MetaAgents: Large Language Model Based Agents for Decision-Making on Teamingcited by 85
  4. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  5. 2024Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMcited by 77
  6. 2023Automated Annotation with Generative AI Requires Validationcited by 75
  7. 2024Self-Evolving Multi-Agent Collaboration Networks for Software Developmentcited by 73
  8. 2023Questioning the Survey Responses of Large Language Modelscited by 70
  9. 2024Hypothesis Generation with Large Language Modelscited by 68
  10. 2023CompeteAI: Understanding the Competition Dynamics in Large Language Model-based Agentscited by 65
  11. 2024Can Large Language Model Agents Simulate Human Trust Behavior?cited by 62
  12. 2023Explaining black box text modules in natural language with language modelscited by 61
  13. 2023Stance Detection with Collaborative Role-Infused LLM-Based Agentscited by 53
  14. 2024Virtual Personas for Language Models via an Anthology of Backstoriescited by 49
  15. 2023Disinformation Detection: An Evolving Challenge in the Age of LLMscited by 41
  16. 2024Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Contextcited by 41
  17. 2023GenAI Against Humanity: Nefarious Applications of Generative Artificial Intelligence and Large Language Modelscited by 40
  18. 2023Benefits and Harms of Large Language Models in Digital Mental Healthcited by 39
  19. 2023GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLPcited by 37
  20. 2023Generating Faithful Synthetic Data with Large Language Models: A Case Study in Computational Social Sciencecited by 37
  21. 2023Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Modelscited by 37
  22. 2026SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agentscited by 37
  23. 2023Rethinking Machine Ethics -- Can LLMs Perform Moral Reasoning through the Lens of Moral Theories?cited by 36
  24. 2023GameEval: Evaluating LLMs on Conversational Gamescited by 35
  25. 2023ALYMPICS: LLM Agents Meet Game Theory -- Exploring Strategic Decision-Making with AI Agentscited by 33
  26. 2023Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuningcited by 31
  27. 2023Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?cited by 30
  28. 2024AI for social science and social science of AI: A Surveycited by 29
  29. 2024Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Conceptscited by 28
  30. 2025Do Large Language Models Solve the Problems of Agent-Based Modeling? A Critical Review of Generative Social Simulationscited by 28
  31. 2024'Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participantscited by 27
  32. 2024How Susceptible are Large Language Models to Ideological Manipulation?cited by 26
  33. 2023Embedding Democratic Values into Social Media AIs via Societal Objective Functionscited by 25
  34. 2025Sparse Autoencoders for Hypothesis Generationcited by 24
  35. 2023Sociocultural Norm Similarities and Differences via Situational Alignment and Explainable Textual Entailmentcited by 22
  36. 2023Using Large Language Models for Qualitative Analysis can Introduce Serious Biascited by 22
  37. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  38. 2024Evaluating Large Language Models with Psychometricscited by 22

50 most cited of 245 All 245 (JSON)