mlsanityData to 7 Oct 2026

Discovering Language Model Behaviors with Model-Written Evaluations

arXiv 2212.09251 ↗First posted 19 Dec 2022cs.CL cs.AI cs.LGcited by 805

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winogender schemas with multiple stages of LM-based generation and filtering. Crowdworkers rate the examples as highly relevant and agree with 90-100% of labels, sometimes more so than corresponding human-written datasets. We generate 154 datasets and discover new cases of inverse scaling where LMs get worse with size. Larger LMs repeat back a dialog user's preferred answer ("sycophancy") and express greater desire to pursue concerning goals like resource acquisition and goal preservation. We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse. For example, RLHF makes LMs express stronger political views (on gun rights and immigration) and a greater desire to avoid shut down. Overall, LM-written evaluations are high-quality and let us quickly discover many novel LM behaviors.

Key facts. Cited by 805 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 72 of its 105 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

72 of 105 references.

References · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
38 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2020Extracting Training Data from Large Language Modelscited by 2,144
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  6. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  7. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  10. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  13. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  14. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  15. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  16. 2022Red Teaming Language Models with Language Modelscited by 983
  17. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  18. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  19. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  20. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  21. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  22. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  23. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  24. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  25. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  26. 2018Gender Bias in Coreference Resolutioncited by 438
  27. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  28. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  29. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  30. 2018AI safety via debatecited by 384
  31. 2022Self-critiquing models for assisting human evaluatorscited by 372
  32. 2021True Few-Shot Learning with Language Modelscited by 343
  33. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  34. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  35. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  36. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  37. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  38. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 99; 2024: 165; 2025: 196; 2026: 344.

805 papers.

Sort:

By area:

  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  3. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  4. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  5. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  6. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  7. 2023Whose Opinions Do Language Models Reflect?cited by 759
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  10. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  11. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  12. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
38 more
  1. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  2. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  3. 2025Humanity's Last Examcited by 467
  4. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  5. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  6. 2024Alignment faking in large language modelscited by 316
  7. 2023Role-Play with Large Language Modelscited by 290
  8. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  9. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  10. 2023Large Language Model Alignment: A Surveycited by 241
  11. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 211
  12. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 206
  13. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  14. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  15. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  16. 2024LLaGA: Large Language and Graph Assistantcited by 177
  17. 2024A Roadmap to Pluralistic Alignmentcited by 175
  18. 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
  19. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  20. 2023Model evaluation for extreme riskscited by 167
  21. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 166
  22. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 161
  23. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  24. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  25. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  26. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  27. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  28. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 136
  29. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
  30. 2023Taken out of context: On measuring situational awareness in LLMscited by 131
  31. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  32. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
  33. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 130
  34. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  35. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  36. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  37. 2023Evaluating and Mitigating Discrimination in Language Model Decisionscited by 117
  38. 2024Empowering Biomedical Discovery with AI Agentscited by 117

50 most cited of 805 All 805 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition
  2. 2024Evaluating Language Models as Synthetic Data Generators
  3. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
  4. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  5. 2025Accelerating Unbiased LLM Evaluation via Synthetic Feedback
  6. 2024Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration
  7. 2025Pitfalls of Scale: Investigating the Inverse Task of Redefinition in Large Language Models
  8. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation