mlsanityData to 13 Oct 2026

SCRuB: Social Concept Reasoning under Rubric-Based Evaluation

arXiv 2605.06444 ↗First posted 7 May 2026cs.AIcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

While many studies of Large Language Model (LLM) reasoning capabilities emphasize mathematical or technical tasks, few address reasoning about social concepts: the abstract ideas shaping social norms, culture, and institutions. This understudied capability is essential for modern models acting as social agents, yet no systematic evaluation methodology targets it. We introduce SCRuB (Social Concept Reasoning under Rubric-Based Evaluation), a framework designed for this setting of task indeterminacy. Our goal is to measure the degree to which a model reasons about social concepts with the depth and critical rigor of a human expert. SCRuB proceeds in three phases: prompt construction from established sources, response generation by experts and models, and comparative evaluation using a five-dimensional critical thinking rubric. To enable generalization of the pipeline, we introduce a Panel of Disciplinary Perspectives ensemble validated against independent expert judges. We release SCRuBEval (n=4,711 evaluation prompts) and SCRuBAnnotations (300 expert-authored responses and 150 expert comparative judgments from 45 PhD-level scholars). Our results show that frontier models consistently outperform human experts across all five rubric dimensions. Across 1,170 pairwise comparisons, expert judges ranked a model response first in 80.8% of judgments and preferred model responses overall 74.4% of the time. Ultimately, this study provides the first expert-grounded demonstration of evaluation saturation for social concept reasoning: the single-turn exam-style format has reached its ceiling for models and humans alike.

Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 57 of its 172 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

57 of 172 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Beg to Differ: Understanding Reasoning-Answer Misalignment Across Languages27 Dec
  2. 2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomes18 Oct
  3. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models23 Apr
  4. 2023Language Models can be Logical Solvers10 Nov
  5. 2023GLoRE: Evaluating Logical Reasoning of Large Language Models13 Oct
  6. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models2 Oct
  7. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-47 Apr

Other references · 50

  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,999
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,197
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,011
  5. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,870
  6. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,201
  7. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 689
  8. 2025Humanity's Last Examcited by 478
  9. 2018Scalable agent alignment via reward modeling: a research directioncited by 443
  10. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 428
  11. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 404
  12. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 359
38 more
  1. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 345
  2. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 303
  3. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 302
  4. 2019Abductive Commonsense Reasoningcited by 300
  5. 2023AI Alignment: A Comprehensive Surveycited by 261
  6. 2024Self-Preference Bias in LLM-as-a-Judgecited by 253
  7. 2024LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individualscited by 244
  8. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 213
  9. 2023Does Writing with Language Models Reduce Content Diversity?cited by 191
  10. 2024A Roadmap to Pluralistic Alignmentcited by 179
  11. 2022In-context Examples Selection for Machine Translationcited by 172
  12. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 156
  13. 2024The Ethics of Advanced AI Assistantscited by 105
  14. 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 90
  15. 2024Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approachcited by 75
  16. 2020LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoningcited by 69
  17. 2022Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writerscited by 69
  18. 2023Evaluating the Zero-shot Robustness of Instruction-tuned Language Modelscited by 65
  19. 2022Quantifying Social Biases Using Templates is Unreliablecited by 38
  20. 2024LLMs May Perform MCQA by Selecting the Least Incorrect Optioncited by 32
  21. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Datasetcited by 32
  22. 2023LogiCoT: Logical Chain-of-Thought Instruction-Tuningcited by 31
  23. 2022Cultural Incongruencies in Artificial Intelligencecited by 21
  24. 2026Brittlebench: Quantifying LLM robustness via prompt sensitivitycited by 20
  25. 2021Leveraging Expert Consistency to Improve Algorithmic Decision Supportcited by 17
  26. 2025The Measurement Imbalance in Agentic AI Evaluation Undermines Industry Productivity Claimscited by 14
  27. 2025Stress-Testing Model Specs Reveals Character Differences among Language Modelscited by 12
  28. 2025Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quiltcited by 9
  29. 2025LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblancecited by 9
  30. 2025Task-Dependent Evaluation of LLM Output Homogenization: A Taxonomy-Guided Frameworkcited by 8
  31. 2025Agree to Disagree? A Meta-Evaluation of LLM Misgenderingcited by 7
  32. 2025Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in Language Modelscited by 6
  33. 2024Aligning Model Evaluations with Human Preferences: Mitigating Token Count Bias in Language Model Assessmentscited by 3
  34. 2024A Framework for Evaluating LLMs Under Task Indeterminacycited by 3
  35. 2025EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferencescited by 3
  36. 2025BiasCause: Evaluate Socially Biased Causal Reasoning of Large Language Modelscited by 2
  37. 2026No Single Best Model for Diversity: Learning a Router for Sample Diversitycited by 2
  38. 2025Context Misleads LLMs: The Role of Context Filtering in Maintaining Safe Alignment of LLMscited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2025SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments
  2. 2023Understanding Social Reasoning in Language Models with Language Models
  3. 2025Social Genome: Grounded Social Reasoning Abilities of Multimodal Models
  4. 2025Investigating Thinking Behaviours of Reasoning-Based Language Models for Social Bias Mitigation
  5. 2024SocialGaze: Improving the Integration of Human Social Norms in Large Language Models
  6. 2024EgoSocialArena: Benchmarking the Social Intelligence of Large Language Models from a First-person Perspective
  7. 2023Examining the Causal Effect of First Names on Language Models: The Case of Social Commonsense Reasoning
  8. 2025To Mask or to Mirror: Human-AI Alignment in Collective Reasoning