A Red Teaming Framework for Securing AI in Maritime Autonomous Systems

arXiv 2312.11500 ↗First posted 8 Dec 2023cs.CR cs.AIcited by 3

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Artificial intelligence (AI) is being ubiquitously adopted to automate processes in science and industry. However, due to its often intricate and opaque nature, AI has been shown to possess inherent vulnerabilities which can be maliciously exploited with adversarial AI, potentially putting AI users and developers at both cyber and physical risk. In addition, there is insufficient comprehension of the real-world effects of adversarial AI and an inadequacy of AI security examinations; therefore, the growing threat landscape is unknown for many AI solutions. To mitigate this issue, we propose one of the first red team frameworks for evaluating the AI security of maritime autonomous systems. The framework provides operators with a proactive (secure by design) and reactive (post-deployment evaluation) response to securing AI technology today and in the future. This framework is a multi-part checklist, which can be tailored to different systems and requirements. We demonstrate this framework to be highly effective for a red team to use to uncover numerous vulnerabilities within a real-world maritime autonomous systems AI, ranging from poisoning to adversarial patch attacks. The lessons learned from systematic AI red teaming can help prevent MAS-related catastrophic events in a world with increasing uptake and reliance on mission-critical AI.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 46 of its 108 references that are in the map. First posted on arXiv 8 Dec 2023.

Builds on

46 of 108 references.

Closest ancestors · 1

Same direction, newest first

2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming13 Nov

Other references · 45

2014Explaining and Harnessing Adversarial Examplescited by 7,364
2013Intriguing properties of neural networkscited by 5,155
2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
2017Ensemble Adversarial Training: Attacks and Defensescited by 840
2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 678
2017One pixel attack for fooling deep neural networkscited by 582
2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 519
2018Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Modelscited by 362
2016DeepSurv: Personalized Treatment Recommender System Using A Cox Proportional Hazards Deep Neural Networkcited by 358
2018Analyzing Federated Learning through an Adversarial Lenscited by 351
2017Defense against Adversarial Attacks Using High-Level Representation Guided Denoisercited by 329
2017Certified Defenses for Data Poisoning Attackscited by 303

33 more

2019On the Detection of Digital Face Manipulationcited by 260
2020The Creation and Detection of Deepfakes: A Surveycited by 247
2018PRADA: Protecting against DNN Model Stealing Attackscited by 165
2018Physical Adversarial Examples for Object Detectorscited by 163
2018DPatch: An Adversarial Patch Attack on Object Detectorscited by 147
2020Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attackscited by 116
2022Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoningcited by 94
2022I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defencescited by 75
2021Segment and Complete: Defending Object Detectors against Adversarial Patch Attacks with Robust Patch Detectioncited by 67
2018Adversarial Malware Binaries: Evading Deep Learning for Malware Detection in Executablescited by 62
2021Exploiting Explanations for Model Inversion Attackscited by 49
2022BadDet: Backdoor Attacks on Object Detectioncited by 45
2020Backdoor Attacks against Transfer Learning with Pre-trained Deep Learning Modelscited by 44
2017Security Evaluation of Pattern Classifiers under Attackcited by 39
2021The Threat of Offensive AI to Organizationscited by 38
2020Satellite Communications in the New Space Era: A Survey and Future Challengescited by 36
2019Role of Spatial Context in Adversarial Robustness for Object Detectioncited by 35
2018Adversarial Deep Learning for Robust Detection of Binary Encoded Malwarecited by 32
2021DetectorGuard: Provably Securing Object Detectors against Localized Patch Hiding Attackscited by 29
2022ImageNet-Patch: A Dataset for Benchmarking Machine Learning Robustness against Adversarial Patchescited by 27
2022Machine Learning Security in Industry: A Quantitative Surveycited by 21
2022Just Rotate it: Deploying Backdoor Attacks via Rotation Transformationcited by 20
2020Arms Race in Adversarial Malware Detection: A Surveycited by 16
2022Poisoning Attacks and Defenses on Artificial Intelligence: A Surveycited by 16
2018Attack Strength vs. Detectability Dilemma in Adversarial Machine Learningcited by 14
2019POBA-GA: Perturbation Optimized Black-Box Adversarial Attacks via Genetic Algorithmcited by 14
2018Defending against Adversarial Images using Basis Functions Transformationscited by 11
2023Attacking by Aligning: Clean-Label Backdoor Attacks on Object Detectioncited by 11
2014Security Evaluation of Support Vector Machines in Adversarial Environmentscited by 8
2018Towards Adversarial Malware Detection: Lessons Learned from PDF-based Attackscited by 5
2019Defending against Machine Learning based Inference Attacks via Adversarial Examples: Opportunities and Challengescited by 3
2022Backdoor Attack Detection in Computer Vision by Applying Matrix Factorization on the Weights of Deep Networkscited by 3
2023AD-YOLO: You Look Only Once in Training Multiple Sound Event Localization and Detectioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

Similar, not cited

Close in wording, no citation either way