mlsanityData to 7 Oct 2026

Fine-tuning a Multiple Instance Learning Feature Extractor with Masked Context Modelling and Knowledge Distillation

arXiv 2403.05325 ↗First posted 8 Mar 2024cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

The first step in Multiple Instance Learning (MIL) algorithms for Whole Slide Image (WSI) classification consists of tiling the input image into smaller patches and computing their feature vectors produced by a pre-trained feature extractor model. Feature extractor models that were pre-trained with supervision on ImageNet have proven to transfer well to this domain, however, this pre-training task does not take into account that visual information in neighboring patches is highly correlated. Based on this observation, we propose to increase downstream MIL classification by fine-tuning the feature extractor model using textitMasked Context Modelling with Knowledge Distillation. In this task, the feature extractor model is fine-tuned by predicting masked patches in a bigger context window. Since reconstructing the input image would require a powerful image generation model, and our goal is not to generate realistically looking image patches, we predict instead the feature vectors produced by a larger teacher network. A single epoch of the proposed task suffices to increase the downstream performance of the feature-extractor model when used in a MIL scenario, even capable of outperforming the downstream performance of the teacher model, while being considerably smaller and requiring a fraction of its compute.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 31 of its 51 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

31 of 51 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  12. 2022Generative Adversarial Networkscited by 5,603
19 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  6. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  7. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  8. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
  9. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 738
  10. 2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 549
  11. 2020Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learningcited by 473
  12. 2021Domain Adaptation for Medical Image Analysis: A Surveycited by 351
  13. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  14. 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
  15. 2021ReSSL: Relational Self-Supervised Learning with Weak Augmentationcited by 83
  16. 2022Adversarial Masking for Self-Supervised Learningcited by 78
  17. 2018Fast Human Pose Estimationcited by 68
  18. 2023Multiple Instance Learning Framework with Masked Hard Instance Mining for Whole Slide Image Classificationcited by 58
  19. 2020Joint Liver Lesion Segmentation and Classification via Transfer Learningcited by 14

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2025Histo-Miner: Deep learning based tissue features extraction pipeline from H&E whole slide images of cutaneous squamous cell carcinomacited by 2
  2. 2024EFCM: Efficient Fine-tuning on Compressed Models for deployment of large models in medical image analysiscited by 0