# Rethinking the Backward Propagation for Adversarial Transferability

arXiv: https://arxiv.org/abs/2306.12685 · HTML: https://mlsanity.com/paper/2306.12685/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 59 references that are in the map.
- First posted on arXiv 22 Jun 2023.

## Builds on (most cited first, 30 of 52)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [The Limitations of Deep Learning in Adversarial Settings](https://mlsanity.com/paper/1511.07528/) (2015)
- [ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models](https://mlsanity.com/paper/1708.03999/) (2017)
- [Improving Transferability of Adversarial Examples with Input Diversity](https://mlsanity.com/paper/1803.06978/) (2018)
- [Delving into Transferable Adversarial Examples and Black-box Attacks](https://mlsanity.com/paper/1611.02770/) (2016)
- [Countering Adversarial Images using Input Transformations](https://mlsanity.com/paper/1711.00117/) (2017)
- [Black-box Adversarial Attacks with Limited Queries and Information](https://mlsanity.com/paper/1804.08598/) (2018)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models](https://mlsanity.com/paper/1712.04248/) (2017)
- [Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks](https://mlsanity.com/paper/1904.02884/) (2019)
- [Mitigating Adversarial Effects Through Randomization](https://mlsanity.com/paper/1711.01991/) (2017)
- [Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks](https://mlsanity.com/paper/1908.06281/) (2019)
- [HopSkipJumpAttack: A Query-Efficient Decision-Based Attack](https://mlsanity.com/paper/1904.02144/) (2019)
- [Defense against Adversarial Attacks Using High-Level Representation Guided Denoiser](https://mlsanity.com/paper/1712.02976/) (2017)

## Built on by (most cited first, 27 of 27)

- [Boosting Adversarial Transferability by Block Shuffle and Rotation](https://mlsanity.com/paper/2308.10299/) (2023)
- [Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferability](https://mlsanity.com/paper/2307.07873/) (2023)
- [Improving the Transferability of Adversarial Examples with Arbitrary Style Transfer](https://mlsanity.com/paper/2308.10601/) (2023)
- [Attention! Your Vision Language Model Could Be Maliciously Manipulated](https://mlsanity.com/paper/2505.19911/) (2025)
- [Improving Adversarial Transferability via Model Alignment](https://mlsanity.com/paper/2311.18495/) (2023)
- [Understanding Model Ensemble in Transferable Adversarial Attack](https://mlsanity.com/paper/2410.06851/) (2024)
- [Improving Adversarial Transferability on Vision Transformers via Forward Propagation Refinement](https://mlsanity.com/paper/2503.15404/) (2025)
- [SoK: Pitfalls in Evaluating Black-Box Attacks](https://mlsanity.com/paper/2310.17534/) (2023)
- [Adversarial Example Soups: Improving Transferability and Stealthiness for Free](https://mlsanity.com/paper/2402.18370/) (2024)
- [Exploring Adversarial Transferability between Kolmogorov-arnold Networks](https://mlsanity.com/paper/2503.06276/) (2025)
- [Rethinking Mixup for Improving the Adversarial Transferability](https://mlsanity.com/paper/2311.17087/) (2023)
- [Advancing Generalized Transfer Attack with Initialization Derived Bilevel Optimization and Dynamic Sequence Truncation](https://mlsanity.com/paper/2406.02064/) (2024)
- [Boosting the Local Invariance for Better Adversarial Transferability](https://mlsanity.com/paper/2503.06140/) (2025)
- [TESSER: Transfer-Enhancing Adversarial Attacks from Vision Transformers via Spectral and Semantic Regularization](https://mlsanity.com/paper/2505.19613/) (2025)
- [Text Adversarial Attacks with Dynamic Outputs](https://mlsanity.com/paper/2509.22393/) (2025)
- [Generating Visually Realistic Adversarial Patch](https://mlsanity.com/paper/2312.03030/) (2023)
- [Network transferability of adversarial patches in real-time object detection](https://mlsanity.com/paper/2408.15833/) (2024)
- [On the Adversarial Transferability of Generalized "Skip Connections"](https://mlsanity.com/paper/2410.08950/) (2024)
- [Boosting Adversarial Transferability with Spatial Adversarial Alignment](https://mlsanity.com/paper/2501.01015/) (2025)
- [Disrupting Semantic and Abstract Features for Better Adversarial Transferability](https://mlsanity.com/paper/2507.16052/) (2025)
- [Multi-task Adversarial Attacks against Black-box Model with Few-shot Queries](https://mlsanity.com/paper/2508.10039/) (2025)
- [ViT-EnsembleAttack: Augmenting Ensemble Models for Stronger Adversarial Transferability in Vision Transformers](https://mlsanity.com/paper/2508.12384/) (2025)
- [Transform-Dependent Adversarial Attacks](https://mlsanity.com/paper/2406.08443/) (2024)
- [Training on Plausible Counterfactuals Removes Spurious Correlations](https://mlsanity.com/paper/2505.16583/) (2025)
- [Temporal Consistency Constrained Transferable Adversarial Attacks with Background Mixup for Action Recognition](https://mlsanity.com/paper/2505.17807/) (2025)
- [Devling into Adversarial Transferability on Image Classification: Review, Benchmark, and Evaluation](https://mlsanity.com/paper/2602.23117/) (2026)
- [Learning with Bilevel-Minimax Optimization for Efficient and Reliable Transfer Attacks](https://mlsanity.com/paper/2608.11815/) (2026)

## Abstract (arXiv)

Transfer-based attacks generate adversarial examples on the surrogate model, which can mislead other black-box models without access, making it promising to attack real-world applications. Recently, several works have been proposed to boost adversarial transferability, in which the surrogate model is usually overlooked. In this work, we identify that non-linear layers (e.g., ReLU, max-pooling, etc.) truncate the gradient during backward propagation, making the gradient w.r.t. input image imprecise to the loss function. We hypothesize and empirically validate that such truncation undermines the transferability of adversarial examples. Based on these findings, we propose a novel method called Backward Propagation Attack (BPA) to increase the relevance between the gradient w.r.t. input image and loss function so as to generate adversarial examples with higher transferability. Specifically, BPA adopts a non-monotonic function as the derivative of ReLU and incorporates softmax with temperature to smooth the derivative of max-pooling, thereby mitigating the information loss during the backward propagation of gradients. Empirical results on the ImageNet dataset demonstrate that not only does our method substantially boost the adversarial transferability, but it is also general to existing transfer-based attacks. Code is available at https://github.com/Trustworthy-AI-Group/RPA.