# Real-World Gaps in AI Governance Research

arXiv: https://arxiv.org/abs/2505.00174 · HTML: https://mlsanity.com/paper/2505.00174/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 120 references that are in the map.
- First posted on arXiv 30 Apr 2025.

## Builds on (most cited first, 30 of 43)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models](https://mlsanity.com/paper/2406.10162/) (2024)
- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [Evaluating Frontier Models for Dangerous Capabilities](https://mlsanity.com/paper/2403.13793/) (2024)
- [Frontier AI Regulation: Managing Emerging Risks to Public Safety](https://mlsanity.com/paper/2307.03718/) (2023)
- [The Fallacy of AI Functionality](https://mlsanity.com/paper/2206.09511/) (2022)
- [Clio: Privacy-Preserving Insights into Real-World AI Use](https://mlsanity.com/paper/2412.13678/) (2024)
- [The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research](https://mlsanity.com/paper/2010.15581/) (2020)
- [Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs](https://mlsanity.com/paper/2503.08688/) (2025)
- [Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactions](https://mlsanity.com/paper/2504.15236/) (2025)
- [A Safe Harbor for AI Evaluation and Red Teaming](https://mlsanity.com/paper/2403.04893/) (2024)
- [Toward an Evaluation Science for Generative AI Systems](https://mlsanity.com/paper/2503.05336/) (2025)
- [The Grey Hoodie Project: Big Tobacco, Big Tech, and the threat on academic integrity](https://mlsanity.com/paper/2009.13676/) (2020)
- [Securing Large Language Models: Threats, Vulnerabilities and Responsible Practices](https://mlsanity.com/paper/2403.12503/) (2024)
- [HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wild](https://mlsanity.com/paper/2403.04307/) (2024)
- [FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation](https://mlsanity.com/paper/2410.22257/) (2024)
- [Lessons From Red Teaming 100 Generative AI Products](https://mlsanity.com/paper/2501.07238/) (2025)
- [Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data](https://mlsanity.com/paper/2406.13843/) (2024)

## Built on by (most cited first, 5 of 5)

- [Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects](https://mlsanity.com/paper/2505.18893/) (2025)
- [Statistical Methods in Generative AI](https://mlsanity.com/paper/2509.07054/) (2025)
- [A Maslow-Inspired Hierarchy of Engagement with AI Model](https://mlsanity.com/paper/2509.07032/) (2025)
- [Studying People to Study AI: Expert Perspectives on the Epistemic Fit and Barriers of Human Research in AI Safety & Ethics](https://mlsanity.com/paper/2608.05656/) (2026)
- [Context-Aware Pre-Deployment Evaluation of AI Systems: A Regulatory Framework for Nigerian Fintech](https://mlsanity.com/paper/2609.24016/) (2026)

## Abstract (arXiv)

Drawing on 1,178 safety and reliability papers from 9,439 generative AI papers (January 2020 - March 2025), we compare research outputs of leading AI companies (Anthropic, Google DeepMind, Meta, Microsoft, and OpenAI) and AI universities (CMU, MIT, NYU, Stanford, UC Berkeley, and University of Washington). We find that corporate AI research increasingly concentrates on pre-deployment areas -- model alignment and testing & evaluation -- while attention to deployment-stage issues such as model bias has waned. Significant research gaps exist in high-risk deployment domains, including healthcare, finance, misinformation, persuasive and addictive features, hallucinations, and copyright. Without improved observability into deployed AI, growing corporate concentration could deepen knowledge deficits. We recommend expanding external researcher access to deployment data and systematic observability of in-market AI behaviors.