# Governing Cloud Data Pipelines with Agentic AI

arXiv: https://arxiv.org/abs/2512.23737 · HTML: https://mlsanity.com/paper/2512.23737/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 2 of its 28 references that are in the map.
- First posted on arXiv 24 Dec 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 2 of 2)

- [LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services](https://mlsanity.com/paper/2410.02425/) (2024)
- [A Privacy-Preserving Cloud Architecture for Distributed Machine Learning at Scale](https://mlsanity.com/paper/2512.10341/) (2025)

## Built on by (most cited first, 3 of 3)

- [AI-Driven Cloud Resource Optimization for Multi-Cluster Environments](https://mlsanity.com/paper/2512.24914/) (2025)
- [Cognitive Platform Engineering for Autonomous Cloud Operations](https://mlsanity.com/paper/2601.17542/) (2026)
- [Graph, Loop, and Harness Engineering for Zero-Trust Agentic Data Engineering and Analytical Processing](https://mlsanity.com/paper/2609.29668/) (2026)

## Abstract (arXiv)

Cloud data pipelines increasingly operate under dynamic workloads, evolving schemas, cost constraints, and strict governance requirements. Despite advances in cloud-native orchestration frameworks, most production pipelines rely on static configurations and reactive operational practices, resulting in prolonged recovery times, inefficient resource utilization, and high manual overhead. This paper presents Agentic Cloud Data Engineering, a policy-aware control architecture that integrates bounded AI agents into the governance and control plane of cloud data pipelines. In Agentic Cloud Data Engineering platform, specialized agents analyze pipeline telemetry and metadata, reason over declarative cost and compliance policies, and propose constrained operational actions such as adaptive resource reconfiguration, schema reconciliation, and automated failure recovery. All agent actions are validated against governance policies to ensure predictable and auditable behavior. We evaluate Agentic Cloud Data Engineering platform using representative batch and streaming analytics workloads constructed from public enterprise-style datasets. Experimental results show that Agentic Cloud Data Engineering platform reduces mean pipeline recovery time by up to 45%, lowers operational cost by approximately 25%, and decreases manual intervention events by over 70% compared to static orchestration, while maintaining data freshness and policy compliance. These results demonstrate that policy-bounded agentic control provides an effective and practical approach for governing cloud data pipelines in enterprise environments.