# SCRuB: Social Concept Reasoning under Rubric-Based Evaluation

arXiv: https://arxiv.org/abs/2605.06444 · HTML: https://mlsanity.com/paper/2605.06444/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 57 of its 172 references that are in the map.
- First posted on arXiv 7 May 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 57)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://mlsanity.com/paper/2410.02736/) (2024)
- [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [Self-Preference Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2410.21819/) (2024)
- [LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals](https://mlsanity.com/paper/2411.10109/) (2024)
- [ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning](https://mlsanity.com/paper/2002.04326/) (2020)
- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)
- [Does Writing with Language Models Reduce Content Diversity?](https://mlsanity.com/paper/2309.05196/) (2023)
- [A Roadmap to Pluralistic Alignment](https://mlsanity.com/paper/2402.05070/) (2024)
- [In-context Examples Selection for Machine Translation](https://mlsanity.com/paper/2212.02437/) (2022)
- [BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval](https://mlsanity.com/paper/2407.12883/) (2024)
- [The Ethics of Advanced AI Assistants](https://mlsanity.com/paper/2404.16244/) (2024)
- [Verbosity Bias in Preference Labeling by Large Language Models](https://mlsanity.com/paper/2310.10076/) (2023)
- [Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach](https://mlsanity.com/paper/2404.15993/) (2024)
- [LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning](https://mlsanity.com/paper/2007.08124/) (2020)
- [Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writers](https://mlsanity.com/paper/2211.05030/) (2022)

## Built on by (most cited first, 1 of 1)

- [From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape](https://mlsanity.com/paper/2606.08625/) (2026)

## Abstract (arXiv)

While many studies of Large Language Model (LLM) reasoning capabilities emphasize mathematical or technical tasks, few address reasoning about social concepts: the abstract ideas shaping social norms, culture, and institutions. This understudied capability is essential for modern models acting as social agents, yet no systematic evaluation methodology targets it. We introduce SCRuB (Social Concept Reasoning under Rubric-Based Evaluation), a framework designed for this setting of task indeterminacy. Our goal is to measure the degree to which a model reasons about social concepts with the depth and critical rigor of a human expert. SCRuB proceeds in three phases: prompt construction from established sources, response generation by experts and models, and comparative evaluation using a five-dimensional critical thinking rubric. To enable generalization of the pipeline, we introduce a Panel of Disciplinary Perspectives ensemble validated against independent expert judges. We release SCRuBEval (n=4,711 evaluation prompts) and SCRuBAnnotations (300 expert-authored responses and 150 expert comparative judgments from 45 PhD-level scholars). Our results show that frontier models consistently outperform human experts across all five rubric dimensions. Across 1,170 pairwise comparisons, expert judges ranked a model response first in 80.8% of judgments and preferred model responses overall 74.4% of the time. Ultimately, this study provides the first expert-grounded demonstration of evaluation saturation for social concept reasoning: the single-turn exam-style format has reached its ceiling for models and humans alike.