mlsanityData to 7 Oct 2026

Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do

arXiv 2103.11790 ↗First posted 8 Mar 2021cs.CL cs.CYcited by 178

Not on the map. Why?

Abstract (arXiv)

Artificial writing is permeating our lives due to recent advances in large-scale, transformer-based language models (LMs) such as BERT, its variants, GPT-2/3, and others. Using them as pre-trained models and fine-tuning them for specific tasks, researchers have extended state of the art for many NLP tasks and shown that they capture not only linguistic knowledge but also retain general knowledge implicitly present in the data. Unfortunately, LMs trained on unfiltered text corpora suffer from degenerated and biased behaviour. While this is well established, we show that recent LMs also contain human-like biases of what is right and wrong to do, some form of ethical and moral norms of the society -- they bring a "moral direction" to surface. That is, we show that these norms can be captured geometrically by a direction, which can be computed, e.g., by a PCA, in the embedding space, reflecting well the agreement of phrases to social norms implicitly expressed in the training texts and providing a path for attenuating or even preventing toxic degeneration in LMs. Being able to rate the (non-)normativity of arbitrary phrases without explicitly training the LM for this task, we demonstrate the capabilities of the "moral direction" for guiding (even other) LMs towards producing normative text and showcase it on RealToxicityPrompts testbed, preventing the neural toxic degeneration in GPT-2.

Key facts. Cited by 178 papers in the map (data through 7 Oct 2026). Builds on 35 of its 75 references that are in the map. First posted on arXiv 8 Mar 2021.

Builds on

35 of 75 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2019Language Models as Knowledge Bases?cited by 1,922
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  9. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  11. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  12. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
23 more
  1. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  2. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  3. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  4. 2020Aligning AI With Shared Human Valuescited by 706
  5. 2018Universal Sentence Encodercited by 485
  6. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  7. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  8. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  9. 2017Right for the Right Reasons: Training Differentiable Models by Constraining their Explanationscited by 293
  10. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  11. 2019Measuring Bias in Contextualized Word Representationscited by 275
  12. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 253
  13. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  14. 2019Assessing BERT's Syntactic Abilitiescited by 191
  15. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  16. 2020Making deep neural networks right for the right scientific reasons by interacting with their explanationscited by 141
  17. 2019Assessing Social and Intersectional Biases in Contextualized Word Representationscited by 134
  18. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
  19. 2019Semantics-aware BERT for Language Understandingcited by 111
  20. 2019Gmail Smart Compose: Real-Time Assisted Writingcited by 97
  21. 2019Still a Pain in the Neck: Evaluating Text Representations on Lexical Compositioncited by 27
  22. 2019BERT has a Moral Compass: Improvements of ethical and moral values of machinescited by 21
  23. 2020Reducing Non-Normative Text Generation from Language Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 15; 2023: 51; 2024: 58; 2025: 38; 2026: 15.

178 papers.

Sort:

By area:

  1. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  2. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
  3. 2022Using cognitive psychology to understand GPT-3cited by 389
  4. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 308
  5. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  6. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
  7. 2023Gender bias and stereotypes in Large Language Modelscited by 185
  8. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 183
  9. 2023GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Modelscited by 174
  10. 2023Personality Traits in Large Language Modelscited by 166
  11. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 166
  12. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
38 more
  1. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  2. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 150
  3. 2023Fair Diffusion: Instructing Text-to-Image Generation Models on Fairnesscited by 149
  4. 2021Can Machines Learn Morality? The Delphi Experimentcited by 140
  5. 2022Automatic Generation of Programming Exercises and Code Explanations using Large Language Modelscited by 124
  6. 2023Generative AIcited by 119
  7. 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 116
  8. 2024Cognitive Bias in Decision-Making with LLMscited by 104
  9. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 97
  10. 2024Measuring Political Bias in Large Language Models: What Is Said and How It Is Saidcited by 89
  11. 2023Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Reviewcited by 83
  12. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  13. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  14. 2024Nicer Than Humans: How do Large Language Models Behave in the Prisoner's Dilemma?cited by 62
  15. 2024Style Vectors for Steering Generative Large Language Modelcited by 56
  16. 2023Large Language Models in Education: Vision and Opportunitiescited by 54
  17. 2023Knowledge of cultural moral norms in large language modelscited by 51
  18. 2026Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agentscited by 49
  19. 2024Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Informationcited by 48
  20. 2024Language Model Alignment in Multilingual Trolley Problemscited by 46
  21. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Modelscited by 45
  22. 2024Emergence of Social Norms in Generative Agent Societies: Principles and Architecturecited by 41
  23. 2023GenAI Against Humanity: Nefarious Applications of Generative Artificial Intelligence and Large Language Modelscited by 40
  24. 2024Moral Alignment for LLM Agentscited by 39
  25. 2022Speaking Multiple Languages Affects the Moral Bias of Language Modelscited by 34
  26. 2022Does Moral Code Have a Moral Code? Probing Delphi's Moral Philosophycited by 31
  27. 2022The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarkscited by 31
  28. 2024PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agentscited by 28
  29. 2024Democratizing Fine-grained Visual Recognition with Large Language Modelscited by 25
  30. 2023An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Modelscited by 24
  31. 2024Human-like object concept representations emerge naturally in multimodal large language modelscited by 24
  32. 2023ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Modelscited by 23
  33. 2023Probing the Moral Development of Large Language Models through Defining Issues Testcited by 23
  34. 2024(Ir)rationality and Cognitive Biases in Large Language Modelscited by 23
  35. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  36. 2024DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subculturescited by 23
  37. 2024Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Promptingcited by 22
  38. 2024Improving Context-Aware Preference Modeling for Language Modelscited by 22

50 most cited of 178 All 178 (JSON)