Token-Based Detection of Spurious Correlations in Vision Transformers

Due to their powerful feature association capabilities, neural network-based computer vision models have the ability to detect and exploit unintended patterns within the data, potentially leading to correct predictions based on incorrect or unintended but statistically relevant signals. These clues may vary from simple color aberrations to small pieces of text within the image. In situations where these unintended signals align with the predictive task, models can mistakenly link these features with the task and rely on them for making predictions. This phenomenon is referred to as spurious correlations, where patterns appear to be associated with the task but are actually coincidental. As a result, detection and mitigation of spurious correlations have become crucial tasks for building trustworthy, reliable, and generalizable machine learning models. In this work, we present a token-based diagnostic pipeline that applies leave-one-out token removal to detect spurious correlations in vision transformers. The proposed approach quantifies a model's reliance on non-core visual cues through complementary measures that capture both aggregate and localized spurious effects at the token level. Using both supervised and self-supervised trained models, we present large-scale experiments on the ImageNet dataset demonstrating the ability of the proposed method to identify spurious correlations. We also find that, even if the same architecture is used, the training methodology has a substantial impact on the model's reliance on spurious correlations. Furthermore, we show that for certain ImageNet classes, many images exhibit strong reliance on non-core visual cues across multiple models, and we discuss common sources of such signals (e.g., watermarks and background artifacts). Lastly, we present a case study investigating spurious signals in invasive breast mass classification.