# CRISP: Cultural Reward Modeling for Implicit Situated Propriety

arXiv: https://arxiv.org/abs/2609.34345 · HTML: https://mlsanity.com/paper/2609.34345/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 27 of 27)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts](https://mlsanity.com/paper/2406.12845/) (2024)
- [Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs](https://mlsanity.com/paper/2410.18451/) (2024)
- [Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy](https://mlsanity.com/paper/2507.01352/) (2025)
- [HelpSteer2-Preference: Complementing Ratings with Preferences](https://mlsanity.com/paper/2410.01257/) (2024)
- [Quantile Regression for Distributional Reward Models in RLHF](https://mlsanity.com/paper/2409.10164/) (2024)
- [Knowledge of cultural moral norms in large language models](https://mlsanity.com/paper/2306.01857/) (2023)
- [Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense](https://mlsanity.com/paper/2405.04655/) (2024)
- [M-RewardBench: Evaluating Reward Models in Multilingual Settings](https://mlsanity.com/paper/2410.15522/) (2024)
- [Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors](https://mlsanity.com/paper/2407.09136/) (2024)
- [RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment](https://mlsanity.com/paper/2412.13746/) (2024)
- [Language Models in Dialogue: Conversational Maxims for Human-AI Interactions](https://mlsanity.com/paper/2403.15115/) (2024)
- [Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents](https://mlsanity.com/paper/2506.21252/) (2025)
- [Cultural Learning-Based Culture Adaptation of Language Models](https://mlsanity.com/paper/2504.02953/) (2025)
- [GlobeSumm: A Challenging Benchmark Towards Unifying Multi-lingual, Cross-lingual and Multi-document News Summarization](https://mlsanity.com/paper/2410.04087/) (2024)
- [Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on Korean Superstition](https://mlsanity.com/paper/2507.04014/) (2025)
- [Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation](https://mlsanity.com/paper/2604.24361/) (2026)
- [Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment](https://mlsanity.com/paper/2509.21798/) (2025)
- [x1: Learning to Think Adaptively Across Languages and Cultures](https://mlsanity.com/paper/2604.16917/) (2026)
- [CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs](https://mlsanity.com/paper/2606.01879/) (2026)
- [Steerable Cultural Preference Optimization of Reward Models](https://mlsanity.com/paper/2606.18606/) (2026)

## Abstract (arXiv)

As large language models (LLMs) are increasingly deployed across countries and regions, the ability to recognize and respond appropriately to diverse cultural contexts becomes increasingly important. However, existing research has largely focused on cultural knowledge or tasks with predefined response spaces, while open-ended culturally situated behavior remains comparatively underexplored. In this work, we introduce CRISP-RM, a culturally situated reward model that assigns rewards according to cultural appropriateness in open-ended social scenarios. During policy optimization, we further introduce Norm Grounding Supervision (NGS), providing guidance that enhances the policy's sensitivity to relevant cultural norms. To construct culturally situated data, we employ a collaborative multi-agent framework that instantiates implicit cultural norms into diverse social scenarios and further curate NormCompass as a dedicated testbed. We conduct comprehensive experiments to evaluate the effectiveness of CRISP-RM in both reward modeling and policy optimization. Best-of-(N) experiments show that CRISP-RM consistently outperforms strong general reward models. During GRPO policy optimization, CRISP-RM generally improves culturally situated behavior, while incorporating NGS yields further gains. Further analyses demonstrate the advantages of CRISP-RM in distinguishing culturally appropriate behavior beyond superficial fluency and politeness, while NGS provides complementary gains during policy optimization by improving norm grounding.