mlsanityData to 7 Oct 2026

Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark

arXiv 2305.14938 ↗First posted 24 May 2023cs.CL cs.AIcited by 62

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have been shown to perform well at a variety of syntactic, discourse, and reasoning tasks. While LLMs are increasingly deployed in many forms including conversational agents that interact with humans, we lack a grounded benchmark to measure how well LLMs understand textitsocial language. Here, we introduce a new theory-driven benchmark, SocKET, that contains 58 NLP tasks testing social knowledge which we group into five categories: humor & sarcasm, offensiveness, sentiment & emotion, and trustworthiness. In tests on the benchmark, we demonstrate that current models attain only moderate performance but reveal significant potential for task transfer among different types and categories of tasks, which were predicted from theory. Through zero-shot evaluations, we show that pretrained models already possess some innate but limited capabilities of social language understanding and training on one category of tasks can improve zero-shot testing on others. Our benchmark provides a systematic way to analyze model performance on an important dimension of language and points to clear room for improvement to build more socially-aware LLMs. The associated resources are released at https://github.com/minjechoi/SOCKET.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 61 of its 172 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

61 of 172 references.

References · 61

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
38 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  5. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  6. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  7. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  8. 2019Predicting the Type and Target of Offensive Posts in Social Mediacited by 268
  9. 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 216
  10. 2018Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfercited by 209
  11. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 209
  12. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 206
  13. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  14. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 194
  15. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  16. 2020A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Supportcited by 182
  17. 2021Can Machines Learn Morality? The Delphi Experimentcited by 140
  18. 2017Understanding Abuse: A Typology of Abusive Language Detection Subtaskscited by 115
  19. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  20. 2019Automatically Neutralizing Subjective Bias in Textcited by 105
  21. 2013A Computational Approach to Politeness with Application to Social Factorscited by 104
  22. 2020The Effect of Natural Distribution Shift on Question Answering Modelscited by 104
  23. 2019Evaluating Commonsense in Pre-trained Language Modelscited by 95
  24. 2020Toxicity Detection: Does Context Really Matter?cited by 93
  25. 2020SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articlescited by 79
  26. 2022EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysiscited by 75
  27. 2022A fine-grained comparison of pragmatic language understanding in humans and language modelscited by 69
  28. 2017A Large Self-Annotated Corpus for Sarcasmcited by 63
  29. 2018Modeling Empathy and Distress in Reaction to News Storiescited by 52
  30. 2019A Just and Comprehensive Strategy for Using NLP to Address Online Abusecited by 49
  31. 2020Targeting the Benchmark: On Methodology in Current Natural Language Processing Researchcited by 44
  32. 2021Detecting Cross-Geographic Biases in Toxicity Modeling on Social Mediacited by 31
  33. 2021Measuring Sentence-Level and Aspect-Level (Un)certainty in Science Communicationscited by 27
  34. 2019Automatically Identifying Complaints in Social Mediacited by 25
  35. 2020Quantifying Intimacy in Languagecited by 25
  36. 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 25
  37. 2021Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexicacited by 23
  38. 2019TalkDown: A Corpus for Condescension Detection in Contextcited by 21

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 28; 2025: 11; 2026: 8.

62 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  4. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  5. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  6. 2023You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instrumentscited by 48
  7. 2023Welfare Diplomacy: Benchmarking Language Model Cooperationcited by 43
  8. 2023Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMscited by 36
  9. 2023DialogBench: Evaluating LLMs as Human-like Dialogue Systemscited by 31
  10. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  11. 2023HARE: Explainable Hate Speech Detection with Step-by-Step Reasoningcited by 30
  12. 2024SPRIG: Improving Large Language Model Performance by System Prompt Optimizationcited by 28
38 more
  1. 2024MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platformscited by 26
  2. 2023FFT: Towards Harmlessness Evaluation and Analysis for LLMs with Factuality, Fairness, Toxicitycited by 22
  3. 2024"A good pun is its own reword": Can Large Language Models Understand Puns?cited by 21
  4. 2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
  5. 2024Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technologycited by 17
  6. 2025SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agentscited by 16
  7. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  8. 2024TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planningcited by 14
  9. 2024CLEAR: Can Language Models Really Understand Causal Graphs?cited by 14
  10. 2025Social World Modelscited by 14
  11. 2024Academically intelligent LLMs are not necessarily socially intelligentcited by 12
  12. 2024The Call for Socially Aware Language Technologiescited by 12
  13. 2023Large Human Language Models: A Need and the Challengescited by 11
  14. 2023How Far Are LLMs from Believable AI? A Benchmark for Evaluating the Believability of Human Behavior Simulationcited by 11
  15. 2024SoMeLVLM: A Large Vision Language Model for Social Media Processingcited by 11
  16. 2024Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarizationcited by 11
  17. 2024Social Intelligence Data Infrastructure: Structuring the Present and Navigating the Futurecited by 10
  18. 2024GameTraversalBenchmark: Evaluating Planning Abilities Of Large Language Models Through Traversing 2D Game Mapscited by 8
  19. 2023On Sarcasm Detection with OpenAI GPT-based Modelscited by 7
  20. 2024SOCIALITE-LLAMA: An Instruction-Tuned Model for Social Scientific Taskscited by 7
  21. 2024Polarization of Autonomous Generative AI Agents Under Echo Chamberscited by 7
  22. 2024Decoding News Narratives: A Critical Analysis of Large Language Models in Framing Detectioncited by 6
  23. 2024Prompt Refinement or Fine-tuning? Best Practices for using LLMs in Computational Social Science Taskscited by 6
  24. 2024Comprehensive Reassessment of Large-Scale Evaluation Outcomes in LLMs: A Multifaceted Statistical Approachcited by 5
  25. 2024Comparing Pre-trained Human Language Models: Is it Better with Human Context as Groups, Individual Traits, or Both?cited by 4
  26. 2024Language Evolution for Evading Social Media Regulation via LLM-based Multi-agent Simulationcited by 4
  27. 2024MIKO: Multimodal Intention Knowledge Distillation from Large Language Models for Social-Media Commonsense Discoverycited by 3
  28. 2024In-Memory Learning: A Declarative Learning Framework for Large Language Modelscited by 3
  29. 2024Can large language models understand uncommon meanings of common words?cited by 3
  30. 2025RedditESS: A Mental Health Social Support Interaction Dataset -- Understanding Effective Social Support to Refine AI-Driven Support Toolscited by 3
  31. 2026From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLPcited by 3
  32. 2026Failure of contextual invariance in large language modelscited by 3
  33. 2024Modeling Empathetic Alignment in Conversationcited by 2
  34. 2024ExpressivityBench: Can LLMs Communicate Implicitly?cited by 2
  35. 2025Adaptive Prompting: Ad-hoc Prompt Composition for Social Bias Detectioncited by 2
  36. 2025Agreeing to Interact in Human-Robot Interaction using Large Language Models and Vision Language Modelscited by 2
  37. 2024Binary Hypothesis Testing for Softmax Models and Leverage Score Modelscited by 1
  38. 2024Is the Pope Catholic? Yes, the Pope is Catholic. Generative Evaluation of Non-Literal Intent Resolution in LLMscited by 1

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SI-Bench: Benchmarking Social Intelligence of Large Language Models in Human-to-Human Conversations
  2. 2023Understanding Social Reasoning in Language Models with Language Models
  3. 2025How Social is It? A Benchmark for LLMs' Capabilities in Multi-user Multi-turn Social Agent Tasks
  4. 2025Are they lovers or friends? Evaluating LLMs' Social Reasoning in English and Korean Dialogues
  5. 2025SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments
  6. 2024I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBench
  7. 2024EgoSocialArena: Benchmarking the Social Intelligence of Large Language Models from a First-person Perspective
  8. 2026SocialVeil: Probing Social Intelligence of Language Agents under Communication Barriers