# Modeling Community Attitude through Reaction Tone: A Human-AI Collaborative Framework for Evaluating LLM Alignment with Linguistic Behaviors in Online Communities

arXiv: https://arxiv.org/abs/2605.27388 · HTML: https://mlsanity.com/paper/2605.27388/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 12 Apr 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 26 of 26)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)
- [AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders](https://mlsanity.com/paper/2501.17148/) (2025)
- [AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society](https://mlsanity.com/paper/2502.08691/) (2025)
- [Towards Measuring and Modeling "Culture" in LLMs: A Survey](https://mlsanity.com/paper/2403.15412/) (2024)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [Cultural Bias and Cultural Alignment of Large Language Models](https://mlsanity.com/paper/2311.14096/) (2023)
- [Large language models that replace human participants can harmfully misportray and flatten identity groups](https://mlsanity.com/paper/2402.01908/) (2024)
- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensions](https://mlsanity.com/paper/2309.12342/) (2023)
- [CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies](https://mlsanity.com/paper/2404.15238/) (2024)
- [Unsupervised Keyphrase Extraction with Multipartite Graphs](https://mlsanity.com/paper/1803.08721/) (2018)
- [An Evaluation of Cultural Value Alignment in LLM](https://mlsanity.com/paper/2504.08863/) (2025)
- [Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art](https://mlsanity.com/paper/2406.03930/) (2024)
- [Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs](https://mlsanity.com/paper/2502.08045/) (2025)
- [Whose Emotions and Moral Sentiments Do Language Models Reflect?](https://mlsanity.com/paper/2402.11114/) (2024)
- [Cultural Learning-Based Culture Adaptation of Language Models](https://mlsanity.com/paper/2504.02953/) (2025)
- [Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges](https://mlsanity.com/paper/2507.19672/) (2025)
- [Building Knowledge-Guided Lexica to Model Cultural Variation](https://mlsanity.com/paper/2406.11622/) (2024)
- [Revisiting LLM Value Probing Strategies: Are They Robust and Expressive?](https://mlsanity.com/paper/2507.13490/) (2025)
- [Community Archetypes: An Empirical Framework for Guiding Research Methodologies to Reflect User Experiences of Sense of Virtual Community](https://mlsanity.com/paper/2310.02515/) (2023)
- [Characterizing Similarities and Divergences in Conversational Tones in Humans and LLMs by Sampling with People](https://mlsanity.com/paper/2406.04278/) (2024)
- [Improving and Assessing the Fidelity of Large Language Models Alignment to Online Communities](https://mlsanity.com/paper/2408.09366/) (2024)
- [CommunityBench: Benchmarking Community-Level Alignment across Diverse Groups and Tasks](https://mlsanity.com/paper/2601.13669/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used as proxies for computational social analysis, yet faithfully representing the "thick descriptions" (Geertz, 1973) of human communities remains a critical challenge. Current evaluations often reduce social identity to static labels, sidelining how real-world groups navigate social shifts. To bridge this gap, we introduce CARE (Community-Aware Reaction Evaluation), a reaction-centered framework that benchmarks LLM-simulated discourse against the authentic, event-contingent responses of distinct communities to real-world news. Spanning 207 Reddit communities and covering 9,947 authentic reactions towards 2,166 news articles, CARE evaluates leading LLMs using a hierarchical taxonomy covering coarse attitudes and fine-grained communicative tones. Our empirical findings expose two critical failure modes in prevailing community-conditioning paradigms. First, while community context and targeted reasoning significantly enhance macro-level attitudinal and tonal profiling, these gains largely collapse at the instance level when predicting reactions to specific events. Second, the benefits of community conditioning are remarkably uneven: prompting strategies yield non-uniform shifts, where fidelity gains in some communities are offset by performance drops in others. This micro-macro divergence and community-level instability demonstrate that standard conditioning enables models to approximate static baseline profiles without capturing dynamic or equitable event reactions, establishing CARE as an essential diagnostic tool for community-aware social simulation. Our code and data are available at https://github.com/nuankw/Handle_with_CARE.