Hidden Activations Are Not Enough: A General Approach to Neural Network Predictions
Not on the map. Why?
Abstract (arXiv)
We introduce a novel mathematical framework for analyzing neural networks using tools from quiver representation theory. This framework enables us to quantify the similarity between a new data sample and the training data, as perceived by the neural network. By leveraging the induced quiver representation of a data sample, we capture more information than traditional hidden layer outputs. This quiver representation abstracts away the complexity of the computations of the forward pass into a single matrix, allowing us to employ simple geometric and statistical arguments in a matrix space to study neural network predictions. Our mathematical results are architecture-agnostic and task-agnostic, making them broadly applicable. As proof of concept experiments, we apply our results for the MNIST and FashionMNIST datasets on the problem of detecting adversarial examples on different MLP architectures and several adversarial attack methods. Our experiments can be reproduced with our hrefhttps://github.com/MarcoArmenta/Hidden-Activations-are-not-Enoughpublicly available repository.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 42 references that are in the map. First posted on arXiv 20 Sep 2024.
Builds on
29 of 42 references.
References · 29
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
- 2016Understanding deep learning requires rethinking generalizationcited by 1,674
- 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
- 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
- 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
- 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,167
- 2022Toy Models of Superpositioncited by 971
17 more
- 2017Ensemble Adversarial Training: Attacks and Defensescited by 840
- 2020Fast is better than free: Revisiting adversarial trainingcited by 727
- 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
- 2017One pixel attack for fooling deep neural networkscited by 582
- 2017Sharp Minima Can Generalize For Deep Netscited by 466
- 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
- 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 355
- 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
- 2021Enhancing the Transferability of Adversarial Attacks through Variance Tuningcited by 244
- 2017On Detecting Adversarial Perturbationscited by 232
- 2020Torchattacks: A PyTorch Repository for Adversarial Attackscited by 158
- 2018SparseFool: a few pixels make a big differencecited by 94
- 2019Identity Crisis: Memorization and Generalization under Extreme Overparameterizationcited by 33
- 2022Pixle: a fast and effective black-box attack based on rearranging pixelscited by 20
- 2020The Representation Theory of Neural Networkscited by 19
- 2023Understanding and Improving Ensemble Adversarial Defensecited by 14
- 2021Double framed moduli spaces of quiver representationscited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.