# NICGSlowDown: Evaluating the Efficiency Robustness of Neural Image Caption Generation Models

arXiv: https://arxiv.org/abs/2203.15859 · HTML: https://mlsanity.com/paper/2203.15859/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks](https://mlsanity.com/paper/1704.01155/) (2017)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [Attention on Attention for Image Captioning](https://mlsanity.com/paper/1908.06954/) (2019)
- [X-Linear Attention Networks for Image Captioning](https://mlsanity.com/paper/2003.14080/) (2020)
- [EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examples](https://mlsanity.com/paper/1709.04114/) (2017)
- [Spatially Adaptive Computation Time for Residual Networks](https://mlsanity.com/paper/1612.02297/) (2016)
- [Decoupling Direction and Norm for Efficient Gradient-Based L2 Adversarial Attacks and Defenses](https://mlsanity.com/paper/1811.09600/) (2018)
- [Feature Distillation: DNN-Oriented JPEG Compression Against Adversarial Examples](https://mlsanity.com/paper/1803.05787/) (2018)
- [A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inference](https://mlsanity.com/paper/2010.02432/) (2020)
- [Semantic Compositional Networks for Visual Captioning](https://mlsanity.com/paper/1611.08002/) (2016)

## Built on by (most cited first, 30 of 50)

- [OverThink: Slowdown Attacks on Reasoning LLMs](https://mlsanity.com/paper/2502.02542/) (2025)
- [Inducing High Energy-Latency of Large Vision-Language Models with Verbose Images](https://mlsanity.com/paper/2401.11170/) (2024)
- [Denial-of-Service Poisoning Attacks against Large Language Models](https://mlsanity.com/paper/2410.10760/) (2024)
- [Sibling-Attack: Rethinking Transferable Adversarial Attacks against Face Recognition](https://mlsanity.com/paper/2303.12512/) (2023)
- [FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Model](https://mlsanity.com/paper/2404.14688/) (2024)
- [Poison-splat: Computation Cost Attack on 3D Gaussian Splatting](https://mlsanity.com/paper/2410.08190/) (2024)
- [DeepPerform: An Efficient Approach for Performance Testing of Resource-Constrained Neural Networks](https://mlsanity.com/paper/2210.05370/) (2022)
- [LLMEffiChecker: Understanding and Testing Efficiency Degradation of Large Language Models](https://mlsanity.com/paper/2210.03696/) (2022)
- [On Efficient Variants of Segment Anything Model: A Survey](https://mlsanity.com/paper/2410.04960/) (2024)
- [Dynamic Transformers Provide a False Sense of Efficiency](https://mlsanity.com/paper/2305.12228/) (2023)
- [Energy-Latency Manipulation of Multi-modal Large Language Models via Verbose Samples](https://mlsanity.com/paper/2404.16557/) (2024)
- [RT-LM: Uncertainty-Aware Resource Management for Real-Time Inference of Language Models](https://mlsanity.com/paper/2309.06619/) (2023)
- [SlothSpeech: Denial-of-service Attack Against Speech Recognition Models](https://mlsanity.com/paper/2306.00794/) (2023)
- [LingoLoop Attack: Trapping MLLMs via Linguistic Context and State Entrapment into Endless Loops](https://mlsanity.com/paper/2506.14493/) (2025)
- [PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Models](https://mlsanity.com/paper/2401.15545/) (2024)
- [The Impact of Uniform Inputs on Activation Sparsity and Energy-Latency Attacks in Computer Vision](https://mlsanity.com/paper/2403.18587/) (2024)
- [Your Compiler is Backdooring Your Model: Understanding and Exploiting Compilation Inconsistency Vulnerabilities in Deep Learning Compilers](https://mlsanity.com/paper/2509.11173/) (2025)
- [RemedyGS: Defend 3D Gaussian Splatting against Computation Cost Attacks](https://mlsanity.com/paper/2511.22147/) (2025)
- [RED: A Systematic Real-Time Scheduling Approach for Robotic Environmental Dynamics](https://mlsanity.com/paper/2308.15368/) (2023)
- [Dynamics-aware Adversarial Attack of Adaptive Neural Networks](https://mlsanity.com/paper/2210.08159/) (2022)
- [R^3: On-device Real-Time Deep Reinforcement Learning for Autonomous Robotics](https://mlsanity.com/paper/2308.15039/) (2023)
- [Energy-Latency Attacks: A New Adversarial Threat to Deep Learning](https://mlsanity.com/paper/2503.04963/) (2025)
- [Protecting Your Video Content: Disrupting Automated Video-based LLM Annotations](https://mlsanity.com/paper/2503.21824/) (2025)
- [On Surjectivity of Neural Networks: Can you elicit any behavior from your model?](https://mlsanity.com/paper/2508.19445/) (2025)
- [SlowFormer: Universal Adversarial Patch for Attack on Compute and Energy Efficiency of Inference Efficient Vision Transformers](https://mlsanity.com/paper/2310.02544/) (2023)
- [VLMInferSlow: Evaluating the Efficiency Robustness of Large Vision-Language Models as a Service](https://mlsanity.com/paper/2506.15755/) (2025)
- [DRAINCODE: Stealthy Energy Consumption Attacks on Retrieval-Augmented Code Generation via Context Poisoning](https://mlsanity.com/paper/2601.20615/) (2026)
- [SlowBA: An efficiency backdoor attack towards VLM-based GUI agents](https://mlsanity.com/paper/2603.08316/) (2026)
- [CorrGAN: Input Transformation Technique Against Natural Corruptions](https://mlsanity.com/paper/2204.08623/) (2022)
- [BitHydra: Towards Bit-flip Inference Cost Attack against Large Language Models](https://mlsanity.com/paper/2505.16670/) (2025)

## Abstract (arXiv)

Neural image caption generation (NICG) models have received massive attention from the research community due to their excellent performance in visual understanding. Existing work focuses on improving NICG model accuracy while efficiency is less explored. However, many real-world applications require real-time feedback, which highly relies on the efficiency of NICG models. Recent research observed that the efficiency of NICG models could vary for different inputs. This observation brings in a new attack surface of NICG models, i.e., An adversary might be able to slightly change inputs to cause the NICG models to consume more computational resources. To further understand such efficiency-oriented threats, we propose a new attack approach, NICGSlowDown, to evaluate the efficiency robustness of NICG models. Our experimental results show that NICGSlowDown can generate images with human-unnoticeable perturbations that will increase the NICG model latency up to 483.86%. We hope this research could raise the community's concern about the efficiency robustness of NICG models.