mlsanityData to 7 Oct 2026

Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets

arXiv 2106.10328 ↗First posted 18 Jun 2021cs.CL cs.CYcited by 179

Not on the map. Why?

Cited by 30 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language models can generate harmful and biased outputs and exhibit undesirable behavior according to a given cultural context. We propose a Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets, an iterative process to significantly change model behavior by crafting and fine-tuning on a dataset that reflects a predetermined set of target values. We evaluate our process using three metrics: quantitative metrics with human evaluations that score output adherence to a target value, toxicity scoring on outputs; and qualitative metrics analyzing the most common word associated with a given social category. Through each iteration, we add additional training dataset examples based on observed shortcomings from evaluations. PALMS performs significantly better on all metrics compared to baseline and control models for a broad range of GPT-3 language model sizes without compromising capability integrity. We find that the effectiveness of PALMS increases with model size. We show that significantly adjusting language model behavior is feasible with a small, hand-curated dataset.

Key facts. Cited by 179 papers in the map (data through 7 Oct 2026). Builds on 16 of its 41 references that are in the map. First posted on arXiv 18 Jun 2021.

Builds on

16 of 41 references.

References · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  4. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  7. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  8. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  9. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  10. 2020Towards Debiasing Sentence Representationscited by 170
  11. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  12. 2021Alignment of Language Agentscited by 167
4 more
  1. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 106
  2. 2020Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligencecited by 100
  3. 2020Learning from others' mistakes: Avoiding dataset biases without modeling themcited by 83
  4. 2020Extending the Machine Learning Abstraction Boundary: A Complex Systems Approach to Incorporate Societal Contextcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 29; 2023: 60; 2024: 44; 2025: 22; 2026: 15.

179 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  7. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  10. 2022Red Teaming Language Models with Language Modelscited by 983
  11. 2021Ethical and social risks of harm from Language Modelscited by 957
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
38 more
  1. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  2. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  3. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  4. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  5. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  6. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  7. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  8. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  9. 2022Prompting GPT-3 To Be Reliablecited by 258
  10. 2023Pretraining Language Models with Human Preferencescited by 245
  11. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  12. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  13. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  14. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  15. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 211
  16. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  17. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  18. 2024A Roadmap to Pluralistic Alignmentcited by 175
  19. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  20. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  21. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  22. 2023Self-Alignment with Instruction Backtranslationcited by 149
  23. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  24. 2023Large Language Models in Law: A Surveycited by 128
  25. 2024Collective Constitutional AI: Aligning a Language Model with Public Inputcited by 120
  26. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  27. 2022The Ghost in the Machine has an American accent: value conflict in GPT-3cited by 105
  28. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 94
  29. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  30. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 90
  31. 2024Measuring Political Bias in Large Language Models: What Is Said and How It Is Saidcited by 89
  32. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  33. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 86
  34. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  35. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Modelscited by 73
  36. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  37. 2022Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Modelscited by 67
  38. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 67

50 most cited of 179 All 179 (JSON)