In Search of the Successful Interpolation: On the Role of Sharpness in CLIP Generalization

textitZero-shot models like CLIP are often fine-tuned on a target dataset to improve its accuracy further, but this can compromise out-of-distribution (OOD) robustness. Robust Fine-Tuning (textttRFT )~citepwortsman2021robust, which interpolates between the textitzero-shot and textitfine-tuned models, has been proposed to address this issue. However, understanding when textttRFT actually improves OOD error remains limited. In this work, we empirically investigate the robustness of textttRFT in CLIP models, with a focus on the textitsharpness of the CLIP model during interpolation. First, we demonstrate that while sharpness may not serve as a reliable indicator for predicting the generalization of modern architectures like CLIP on OOD data, this challenges the conventional belief in the generalization benefits of flat minima in foundation models. However, by examining the role of the textitstraggler layer phenomenon, we show that, unlike overall sharpness, the textitlayer-wise sharpness of textitstraggler layers can reliably capture the generalization performance of interpolated CLIP models on OOD data. Our extensive experiments reveal that textitlayer-wise sharpness correlates with generalization in OOD accuracy for textttRFT. Furthermore, we demonstrate that by inducing sparsity in the textitstraggler layers, we can mitigate the textitfailure mode phenomenon in textttRFT. To the best of our knowledge, this is the first work to study the role of sharpness in the textitsuccess of interpolation in the weight space of CLIP foundation models. Our code is available at urlhttps://github.com/alirezaabdollahpour/CLIPModeConnectivity.