# The Competence Shadow: Theory and Bounds of AI Assistance in Safety Engineering

arXiv: https://arxiv.org/abs/2603.25197 · HTML: https://mlsanity.com/paper/2603.25197/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 34 references that are in the map.
- First posted on arXiv 26 Mar 2026.

## Builds on (most cited first, 11 of 11)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [Human-AI collaboration is not very collaborative yet: A taxonomy of interaction patterns in AI-assisted decision making from a systematic review](https://mlsanity.com/paper/2310.19778/) (2023)
- [Can Large Language Models assist in Hazard Analysis?](https://mlsanity.com/paper/2303.15473/) (2023)
- [Aegis:An Advanced LLM-Based Multi-Agent for Intelligent Functional Safety Engineering](https://mlsanity.com/paper/2410.12475/) (2024)
- [Bending the Automation Bias Curve: A Study of Human and AI-based Decision Making in National Security Contexts](https://mlsanity.com/paper/2306.16507/) (2023)
- [Engaging with AI: How Interface Design Shapes Human-AI Collaboration in High-Stakes Decision-Making](https://mlsanity.com/paper/2501.16627/) (2025)
- [SafetyOps](https://mlsanity.com/paper/2008.04461/) (2020)
- [Safety Factories - a Manifesto](https://mlsanity.com/paper/2509.08285/) (2025)

## Abstract (arXiv)

As AI assistants become integrated into safety engineering workflows for Physical AI systems, a critical question emerges: does AI assistance improve safety analysis quality, or introduce systematic blind spots that surface only through post-deployment incidents? This paper develops a formal framework for AI assistance in safety analysis. We first establish why safety engineering resists benchmark-driven evaluation: safety competence is irreducibly multidimensional, constrained by context-dependent correctness, inherent incompleteness, and legitimate expert disagreement. We formalize this through a five-dimensional competence framework capturing domain knowledge, standards expertise, operational experience, contextual understanding, and judgment. We introduce the competence shadow: the systematic narrowing of human reasoning induced by AI-generated safety analysis. The shadow is not what the AI presents, but what it prevents from being considered. We formalize four canonical human-AI collaboration structures and derive closed-form performance bounds, demonstrating that the competence shadow compounds multiplicatively to produce degradation far exceeding naive additive estimates. The central finding is that AI assistance in safety engineering is a collaboration design problem, not a software procurement decision. The same tool degrades or improves analysis quality depending entirely on how it is used. We derive non-degradation conditions for shadow-resistant workflows and call for a shift from tool qualification toward workflow qualification for trustworthy Physical AI.