Risk Architecture for AI-Native Engineering Teams: An Organizational Framework for Agentic System Governance
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
Engineering management research has produced mature frameworks for software risk: ownership by feature, escalation by severity, and assurance by test coverage. These frameworks implicitly assume deterministic behavior, discrete and auditable change events, and clear component-to-owner mappings. Teams that build and operate agentic AI systems violate all three assumptions at once: outputs are probabilistic, systems take autonomous multi-step actions, and the risk surface mutates silently between deployments. Existing AI risk literature addresses this from above (policy frameworks such as the NIST AI RMF and ISO/IEC 42001) or below (threat taxonomies such as OWASP's agentic AI guidance), but not at the layer where an engineering manager (EM) operates: roles, decision rights, and escalation structures. This paper contributes (i) a seven-dimension profile distinguishing pure software-engineering, hybrid, and AI-native teams; (ii) a six-cluster failure-mode taxonomy including a previously unarticulated cluster, dependency-boundary determinism mismatch; and (iii) a synthetic framework-adequacy methodology scoring how well each profile's risk architecture detects, contains, and escalates a defined scenario set. Because the object of study is framework adequacy rather than human behavior, the evaluation yields derived rather than observed coverage claims. Coverage degrades as teams move from pure software engineering to AI-native operation, monotonically in the median and abruptly in the count of uncovered, high-consequence failures appearing only at the AI-native step. The degradation concentrates in specific failure-mode categories, and the most severe, least-covered failures arise not inside AI-native teams but at the organizational boundary where their probabilistic outputs are consumed by determinism-assuming dependencies.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 5 of its 20 references that are in the map. First posted on arXiv 1 Jul 2026.
Builds on
5 of 20 references.
Closest ancestors · 1
Same direction, newest first
Other references · 4
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2023How is ChatGPT's behavior changing over time?cited by 270
- 2026Capability Minimization as a Safety Primitive: Risk-Aware Causal Gating for Least-Privilege LLM Agentscited by 6
- 2026ToolMenuBench: Benchmarking Tool-Menu Filtering Strategies for Reliable and Efficient LLM Agentscited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025With Great Capabilities Come Great Responsibilities: Introducing the Agentic Risk & Capability Framework for Governing Agentic AI Systems
- 2025A Safety and Security Framework for Real-World Agentic Systems
- 2026Exploring Systems-Thinking Approaches to Loss of Control Risk
- 2026AI Assurance: A Comprehensive Testing Strategy for Enterprise AI Systems
- 2026Agentic AI in Engineering and Manufacturing: Industry Perspectives on Utility, Adoption, Challenges, and Opportunities
- 2026From Determinism to Delegation: AI-Native Software Engineering and the Evolution of the Agentic Engineer
- 2026TrustX Agent Risk Classification Framework (ARC): Risk-Tiering Internally Created Agentic AI Systems
- 2025A Frontier AI Risk Management Framework: Bridging the Gap Between Current AI Practices and Established Risk Management