Pixel-Based Similarities as an Alternative to Neural Data for Improving Convolutional Neural Network Adversarial Robustness

Convolutional Neural Networks (CNNs) excel in many visual tasks but remain susceptible to adversarial attacks-imperceptible perturbations that degrade performance. Prior research reveals that brain-inspired regularizers, derived from neural recordings, can bolster CNN robustness; however, reliance on specialized data limits practical adoption. We revisit a regularizer proposed by Li et al. (2019) that aligns CNN representations with neural representational similarity structures and introduce a data-driven variant. Instead of a neural recording-based similarity, our method computes a pixel-based similarity directly from images. This substitution retains the original biologically motivated loss formulation, preserving its robustness benefits while removing the need for neural measurements or task-specific augmentations. Notably, this data-driven variant provides the same robustness improvements observed with neural data. Our approach is lightweight and integrates easily into standard pipelines. Although we do not surpass cutting-edge specialized defenses, we show that neural representational insights can be leveraged without direct recordings. This underscores the promise of robust yet simple methods rooted in brain-inspired principles, even without specialized data, and raises the possibility that further integrating these insights could push performance closer to human levels without resorting to complex, specialized pipelines.