# Mobile Edge Adversarial Detection for Digital Twinning to the Metaverse with Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2303.10288 · HTML: https://mlsanity.com/paper/2303.10288/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 18 Mar 2023.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality](https://mlsanity.com/paper/1801.02613/) (2018)
- [Detecting Adversarial Samples from Artifacts](https://mlsanity.com/paper/1703.00410/) (2017)
- [All One Needs to Know about Metaverse: A Complete Survey on Technological Singularity, Virtual Ecosystem, and Research Agenda](https://mlsanity.com/paper/2110.05352/) (2021)
- [Adversarial and Clean Data Are Not Twins](https://mlsanity.com/paper/1704.04960/) (2017)
- [Inconspicuous Adversarial Patches for Fooling Image Recognition Systems on Mobile Devices](https://mlsanity.com/paper/2106.15202/) (2021)
- [LanCe: A Comprehensive and Lightweight CNN Defense Methodology against Physical Adversarial Attacks on Embedded Multimedia Applications](https://mlsanity.com/paper/1910.08536/) (2019)
- [A Dynamic Resource Allocation Framework for Synchronizing Metaverse with IoT Service and Data](https://mlsanity.com/paper/2111.00431/) (2021)
- [Unified Resource Allocation Framework for the Edge Intelligence-Enabled Metaverse](https://mlsanity.com/paper/2110.14325/) (2021)
- [Resource Allocation for Mobile Metaverse with the Internet of Vehicles over 6G Wireless Communications: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2209.13425/) (2022)
- [Resource Allocation for Edge Computing in IoT Networks via Reinforcement Learning](https://mlsanity.com/paper/1903.01856/) (2019)
- [Detecting Patch Adversarial Attacks with Image Residuals](https://mlsanity.com/paper/2002.12504/) (2020)
- [Resource Allocation in Uplink NOMA-IoT Networks: A Reinforcement-Learning Approach](https://mlsanity.com/paper/2007.08350/) (2020)
- [Optimal Power Allocation for Rate Splitting Communications with Deep Reinforcement Learning](https://mlsanity.com/paper/2107.00238/) (2021)

## Abstract (arXiv)

Real-time Digital Twinning of physical world scenes onto the Metaverse is necessary for a myriad of applications such as augmented-reality (AR) assisted driving. In AR assisted driving, physical environment scenes are first captured by Internet of Vehicles (IoVs) and are uploaded to the Metaverse. A central Metaverse Map Service Provider (MMSP) will aggregate information from all IoVs to develop a central Metaverse Map. Information from the Metaverse Map can then be downloaded into individual IoVs on demand and be delivered as AR scenes to the driver. However, the growing interest in developing AR assisted driving applications which relies on digital twinning invites adversaries. These adversaries may place physical adversarial patches on physical world objects such as cars, signboards, or on roads, seeking to contort the virtual world digital twin. Hence, there is a need to detect these physical world adversarial patches. Nevertheless, as real-time, accurate detection of adversarial patches is compute-intensive, these physical world scenes have to be offloaded to the Metaverse Map Base Stations (MMBS) for computation. Hence in our work, we considered an environment with moving Internet of Vehicles (IoV), uploading real-time physical world scenes to the MMBSs. We formulated a realistic joint variable optimization problem where the MMSPs' objective is to maximize adversarial patch detection mean average precision (mAP), while minimizing the computed AR scene up-link transmission latency and IoVs' up-link transmission idle count, through optimizing the IoV-MMBS allocation and IoV up-link scene resolution selection. We proposed a Heterogeneous Action Proximal Policy Optimization (HAPPO) (discrete-continuous) algorithm to tackle the proposed problem. Extensive experiments shows HAPPO outperforms baseline models when compared against key metrics.