Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Uplift
Not on the map. Why?
Abstract (arXiv)
Current frontier AI safety evaluations emphasize static benchmarks, third-party annotations, and red-teaming. In this position paper, we argue that AI safety research should focus on human-centered evaluations that measure harmful capability uplift: the marginal increase in a user's ability to cause harm with a frontier model beyond what conventional tools already enable. We frame harmful capability uplift as a core AI safety metric, ground it in prior social science research, and provide concrete methodological guidance for systematic measurement. We conclude with actionable steps for developers, researchers, funders, and regulators to make harmful capability uplift evaluation a standard practice.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 24 of its 58 references that are in the map. First posted on arXiv 6 Mar 2026.
Builds on
24 of 58 references.
References · 24
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
- 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
- 2022Red Teaming Language Models with Language Modelscited by 983
- 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
- 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
- 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
- 2022Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionalscited by 191
- 2020Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systemscited by 138
- 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
12 more
- 2024When combinations of humans and AI are useful: A systematic review and meta-analysiscited by 100
- 2024Curiosity-driven Red-teaming for Large Language Modelscited by 83
- 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
- 2025Auditing language models for hidden objectivescited by 68
- 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 56
- 2022Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Modelscited by 49
- 2024Human-In-the-Loop Software Development Agentscited by 42
- 2024Towards interactive evaluations for interaction harms in human-AI systemscited by 27
- 2024Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learningcited by 23
- 2025Safety by Measurement: A Systematic Literature Review of AI Safety Evaluation Methodscited by 15
- 2021The Great Misalignment Problem in Human Evaluation of NLP Methodscited by 8
- 2022A Test for Evaluating Performance in Human-Computer Systemscited by 8
Built on by
Citing papers per year: 2026: 3.
3 papers.