Semantically-Conditioned Negative Samples for Efficient Contrastive Learning

Negative sampling is a limiting factor w.r.t. the generalization of metric-learned neural networks. We show that uniform negative sampling provides little information about the class boundaries and thus propose three novel techniques for efficient negative sampling: drawing negative samples from (1) the top-k most semantically similar classes, (2) the top-k most semantically similar samples and (3) interpolating between contrastive latent representations to create pseudo negatives. Our experiments on CIFAR-10, CIFAR-100 and Tiny-ImageNet-200 show that our proposed textitSemantically Conditioned Negative Sampling and Latent Mixup lead to consistent performance improvements. In the standard supervised learning setting, on average we increase test accuracy by 1.52% percentage points on CIFAR-10 across various network architectures. In the knowledge distillation setting, (1) the performance of student networks increase by 4.56% percentage points on Tiny-ImageNet-200 and 3.29% on CIFAR-100 over student networks trained with no teacher and (2) 1.23% and 1.72% respectively over a textithard-to-beat baseline (Hinton et al., 2015).