# NormSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly

arXiv: https://arxiv.org/abs/2210.08604 · HTML: https://mlsanity.com/paper/2210.08604/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 66 references that are in the map.
- First posted on arXiv 16 Oct 2022.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [Social Chemistry 101: Learning to Reason about Social and Moral Norms](https://mlsanity.com/paper/2011.00620/) (2020)
- [How Many Data Points is a Prompt Worth?](https://mlsanity.com/paper/2103.08493/) (2021)
- [Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets](https://mlsanity.com/paper/2106.10328/) (2021)
- [The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning](https://mlsanity.com/paper/2205.03401/) (2022)
- [Dialogue Natural Language Inference](https://mlsanity.com/paper/1811.00671/) (2018)
- [Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences](https://mlsanity.com/paper/2012.15738/) (2020)
- [Reframing Human-AI Collaboration for Generating Free-Text Explanations](https://mlsanity.com/paper/2112.08674/) (2021)
- [Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotes](https://mlsanity.com/paper/2008.09094/) (2020)
- [The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems](https://mlsanity.com/paper/2204.03021/) (2022)
- [Event2Mind: Commonsense Inference on Events, Intents, and Reactions](https://mlsanity.com/paper/1805.06939/) (2018)
- [A Word on Machine Ethics: A Response to Jiang et al. (2021)](https://mlsanity.com/paper/2111.04158/) (2021)
- [Acquiring Background Knowledge to Improve Moral Value Prediction](https://mlsanity.com/paper/1709.05467/) (2017)
- [DREAM: Improving Situational QA by First Elaborating the Situation](https://mlsanity.com/paper/2112.08656/) (2021)

## Built on by (most cited first, 30 of 48)

- [Towards Measuring and Modeling "Culture" in LLMs: A Survey](https://mlsanity.com/paper/2403.15412/) (2024)
- [Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data](https://mlsanity.com/paper/2302.12822/) (2023)
- [CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models](https://mlsanity.com/paper/2305.14318/) (2023)
- [Word Embeddings Are Steers for Language Models](https://mlsanity.com/paper/2305.12798/) (2023)
- [Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking](https://mlsanity.com/paper/2402.09369/) (2024)
- [Aligning LLMs with Individual Preferences via Interaction](https://mlsanity.com/paper/2410.03642/) (2024)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense](https://mlsanity.com/paper/2405.04655/) (2024)
- [CultureLLM: Incorporating Cultural Differences into Large Language Models](https://mlsanity.com/paper/2402.10946/) (2024)
- [Persona-DB: Efficient Large Language Model Personalization for Response Prediction with Collaborative Data Refinement](https://mlsanity.com/paper/2402.11060/) (2024)
- [Evaluating Cultural and Social Awareness of LLM Web Agents](https://mlsanity.com/paper/2410.23252/) (2024)
- [CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries](https://mlsanity.com/paper/2501.01282/) (2025)
- [Culture is Not Trivia: Sociocultural Theory for Cultural NLP](https://mlsanity.com/paper/2502.12057/) (2025)
- [IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces](https://mlsanity.com/paper/2404.01854/) (2024)
- [Affective Faces for Goal-Driven Dyadic Communication](https://mlsanity.com/paper/2301.10939/) (2023)
- [Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art](https://mlsanity.com/paper/2406.03930/) (2024)
- [NormDial: A Comparable Bilingual Synthetic Dialog Dataset for Modeling Social Norm Adherence and Violation](https://mlsanity.com/paper/2310.14563/) (2023)
- [CulturePark: Boosting Cross-cultural Understanding in Large Language Models](https://mlsanity.com/paper/2405.15145/) (2024)
- [SocialDial: A Benchmark for Socially-Aware Dialogue Systems](https://mlsanity.com/paper/2304.12026/) (2023)
- [Cultural Commonsense Knowledge for Intercultural Dialogues](https://mlsanity.com/paper/2402.10689/) (2024)
- [RENOVI: A Benchmark Towards Remediating Norm Violations in Socio-Cultural Conversations](https://mlsanity.com/paper/2402.11178/) (2024)
- [CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understanding](https://mlsanity.com/paper/2503.10688/) (2025)
- [Decoding the Silent Majority: Inducing Belief Augmented Social Graph with Large Language Model for Response Forecasting](https://mlsanity.com/paper/2310.13297/) (2023)
- [SafeWorld: Geo-Diverse Safety Alignment](https://mlsanity.com/paper/2412.06483/) (2024)
- [RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding](https://mlsanity.com/paper/2505.14462/) (2025)
- [How Well Do LLMs Identify Cultural Unity in Diversity?](https://mlsanity.com/paper/2408.05102/) (2024)
- [CAReDiO: Cultural Alignment via Representativeness and Distinctiveness Guided Data Optimization](https://mlsanity.com/paper/2504.08820/) (2025)
- [NormMark: A Weakly Supervised Markov Model for Socio-cultural Norm Discovery](https://mlsanity.com/paper/2305.16598/) (2023)
- [Defining a New NLP Playground](https://mlsanity.com/paper/2310.20633/) (2023)
- [Isolating Culture Neurons in Multilingual Large Language Models](https://mlsanity.com/paper/2508.02241/) (2025)

## Abstract (arXiv)

Norm discovery is important for understanding and reasoning about the acceptable behaviors and potential violations in human communication and interactions. We introduce NormSage, a framework for addressing the novel task of conversation-grounded multi-lingual, multi-cultural norm discovery, based on language model prompting and self-verification. NormSAGE leverages the expressiveness and implicit knowledge of the pretrained GPT-3 language model backbone, to elicit knowledge about norms through directed questions representing the norm discovery task and conversation context. It further addresses the risk of language model hallucination with a self-verification mechanism ensuring that the norms discovered are correct and are substantially grounded to their source conversations. Evaluation results show that our approach discovers significantly more relevant and insightful norms for conversations on-the-fly compared to baselines (>10+% in Likert scale rating). The norms discovered from Chinese conversation are also comparable to the norms discovered from English conversation in terms of insightfulness and correctness (<3% difference). In addition, the culture-specific norms are promising quality, allowing for 80% accuracy in culture pair human identification. Finally, our grounding process in norm discovery self-verification can be extended for instantiating the adherence and violation of any norm for a given conversation on-the-fly, with explainability and transparency. NormSAGE achieves an AUC of 95.4% in grounding, with natural language explanation matching human-written quality.