# Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Uplift

arXiv: https://arxiv.org/abs/2603.26676 · HTML: https://mlsanity.com/paper/2603.26676/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 58 references that are in the map.
- First posted on arXiv 6 Mar 2026.

## Builds on (most cited first, 24 of 24)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [OR-Bench: An Over-Refusal Benchmark for Large Language Models](https://mlsanity.com/paper/2405.20947/) (2024)
- [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022)
- [AI Sandbagging: Language Models can Strategically Underperform on Evaluations](https://mlsanity.com/paper/2406.07358/) (2024)
- [Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems](https://mlsanity.com/paper/2001.08298/) (2020)
- [When combinations of humans and AI are useful: A systematic review and meta-analysis](https://mlsanity.com/paper/2405.06087/) (2024)
- [Curiosity-driven Red-teaming for Large Language Models](https://mlsanity.com/paper/2402.19464/) (2024)
- [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024)
- [Auditing language models for hidden objectives](https://mlsanity.com/paper/2503.10965/) (2025)
- [Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?](https://mlsanity.com/paper/2407.21792/) (2024)
- [Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models](https://mlsanity.com/paper/2206.08325/) (2022)
- [Human-In-the-Loop Software Development Agents](https://mlsanity.com/paper/2411.12924/) (2024)
- [Towards interactive evaluations for interaction harms in human-AI systems](https://mlsanity.com/paper/2405.10632/) (2024)
- [Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning](https://mlsanity.com/paper/2412.18693/) (2024)
- [Safety by Measurement: A Systematic Literature Review of AI Safety Evaluation Methods](https://mlsanity.com/paper/2505.05541/) (2025)
- [The Great Misalignment Problem in Human Evaluation of NLP Methods](https://mlsanity.com/paper/2104.05361/) (2021)
- [A Test for Evaluating Performance in Human-Computer Systems](https://mlsanity.com/paper/2206.12390/) (2022)

## Built on by (most cited first, 3 of 3)

- [Safeguard-Conditioned Uplift: Measuring Utility-Risk Frontiers for Dual-Use Biology Assistants](https://mlsanity.com/paper/2607.13039/) (2026)
- [NeurIPS Should Require Reproducibility Standards for Frontier AI Safety Claims](https://mlsanity.com/paper/2605.08192/) (2026)
- [The Safeguard Worked. Is the LLM System Safer?](https://mlsanity.com/paper/2609.00519/) (2026)

## Abstract (arXiv)

Current frontier AI safety evaluations emphasize static benchmarks, third-party annotations, and red-teaming. In this position paper, we argue that AI safety research should focus on human-centered evaluations that measure harmful capability uplift: the marginal increase in a user's ability to cause harm with a frontier model beyond what conventional tools already enable. We frame harmful capability uplift as a core AI safety metric, ground it in prior social science research, and provide concrete methodological guidance for systematic measurement. We conclude with actionable steps for developers, researchers, funders, and regulators to make harmful capability uplift evaluation a standard practice.