mlsanityData to 7 Oct 2026

Intriguing properties of neural networks

arXiv 1312.6199 ↗First posted 21 Dec 2013cs.CV cs.LG cs.NEcited by 5,114

Not on the map. Why?

Cited by 212 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks. While their expressiveness is the reason they succeed, it also causes them to learn uninterpretable solutions that could have counter-intuitive properties. In this paper we report two such properties. First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks. Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous to a significant extend. We can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error. In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.

Key facts. Cited by 5,114 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Dec 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2016: 1; 2017: 0; 2018: 1; 2019: 0; 2020: 0; 2021: 1153; 2022: 1044; 2023: 994; 2024: 865; 2025: 670; 2026: 386.

5,114 papers.

Sort:

By area:

  1. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2021Transformers in Vision: A Surveycited by 1,009
  7. 2022Red Teaming Language Models with Language Modelscited by 983
  8. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  9. 2021Domain Generalization: A Surveycited by 760
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  11. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  12. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
38 more
  1. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  2. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  3. 2022Diffusion Models for Adversarial Purificationcited by 469
  4. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  5. 2021Intriguing Properties of Vision Transformerscited by 446
  6. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  7. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  8. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  9. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  10. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
  11. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  12. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
  13. 2023Are aligned neural networks adversarially aligned?cited by 305
  14. 2023Tool Learning with Foundation Modelscited by 302
  15. 2021DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentationcited by 299
  16. 2021Unsolved Problems in ML Safetycited by 291
  17. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  18. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 276
  19. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
  20. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  21. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  22. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  23. 2021Enhancing the Transferability of Adversarial Attacks through Variance Tuningcited by 242
  24. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  25. 2021Improving Robustness using Generated Datacited by 233
  26. 2021Adversarial Neuron Pruning Purifies Backdoored Deep Modelscited by 230
  27. 2022Diffusion models as plug-and-play priorscited by 230
  28. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  29. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  30. 2023An Overview of Catastrophic AI Riskscited by 214
  31. 2022Extremely Simple Activation Shaping for Out-of-Distribution Detectioncited by 212
  32. 2021Towards Causal Representation Learningcited by 207
  33. 2023Cross-Entropy Loss Functions: Theoretical Analysis and Applicationscited by 206
  34. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 203
  35. 2021Fixing Data Augmentation to Improve Adversarial Robustnesscited by 201
  36. 2021Data Augmentation Can Improve Robustnesscited by 193
  37. 2021Unlearnable Examples: Making Personal Data Unexploitablecited by 188
  38. 2021Are Transformers More Robust Than CNNs?cited by 187

50 most cited of 5,114 All 5,114 (JSON)