Explainability Boosted Anomaly Detection Framework for O-RAN based NextG Networks

The wireless networks have historically faced significant security vulnerabilities, necessitating advanced anomaly detection mechanisms, especially as networks evolve towards 6G and beyond. This study introduces an advanced anomaly detection framework that leverages explainable artificial intelligence to enhance the security of next-generation (NextG) cellular networks. By implementing and evaluating a variety of artificial intelligence models, the framework demonstrates high accuracy and efficient runtime performance in identifying malicious traffic within a realistic Open Radio Access Network (O-RAN) testbed. A key innovation of this work is the integration of post-hoc explainability methods to identify the most critical key performance metrics (KPMs), which enables a significant 80% reduction in dataset complexity without compromising detection accuracy. Additionally, explainability analyses identify several critical attack traffic characteristics, such as protocol type, bandwidth, interval, and duration, to prevent upcoming network attacks. The resulting framework effectively balances computational efficiency, accuracy, and explainability, underscoring its practical applicability for enhancing security in next-generation cellular networks.