# Active Finetuning: Exploiting Annotation Budget in the Pretraining-Finetuning Paradigm

arXiv: https://arxiv.org/abs/2303.14382 · HTML: https://mlsanity.com/paper/2303.14382/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 56 references that are in the map.
- First posted on arXiv 25 Mar 2023.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [Learning Loss for Active Learning](https://mlsanity.com/paper/1905.03677/) (2019)
- [Variational Adversarial Active Learning](https://mlsanity.com/paper/1904.00370/) (2019)
- [Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds](https://mlsanity.com/paper/2109.00179/) (2021)
- [Cost-Effective Active Learning for Deep Image Classification](https://mlsanity.com/paper/1701.03551/) (2017)
- [Contextual Diversity for Active Learning](https://mlsanity.com/paper/2008.05723/) (2020)
- [Contrastive Masked Autoencoders are Stronger Vision Learners](https://mlsanity.com/paper/2207.13532/) (2022)
- [Task-Aware Variational Adversarial Active Learning](https://mlsanity.com/paper/2002.04709/) (2020)
- [Influence Selection for Active Learning](https://mlsanity.com/paper/2108.09331/) (2021)

## Built on by (most cited first, 30 of 36)

- [A comprehensive survey on deep active learning in medical image analysis](https://mlsanity.com/paper/2310.14230/) (2023)
- [Multi-Task Model Merging via Adaptive Weight Disentanglement](https://mlsanity.com/paper/2411.18729/) (2024)
- [Reinforced Refinement with Self-Aware Expansion for End-to-End Autonomous Driving](https://mlsanity.com/paper/2506.09800/) (2025)
- [ActiveAD: Planning-Oriented Active Learning for End-to-End Autonomous Driving](https://mlsanity.com/paper/2403.02877/) (2024)
- [Generalized Coverage for More Robust Low-Budget Active Learning](https://mlsanity.com/paper/2407.12212/) (2024)
- [Active Learning for Vision-Language Models](https://mlsanity.com/paper/2410.22187/) (2024)
- [Uncertainty Herding: One Active Learning Method for All Label Budgets](https://mlsanity.com/paper/2412.20644/) (2024)
- [Post-hoc Probabilistic Vision-Language Models](https://mlsanity.com/paper/2412.06014/) (2024)
- [Fairness Without Harm: An Influence-Guided Active Sampling Approach](https://mlsanity.com/paper/2402.12789/) (2024)
- [Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction Tuning](https://mlsanity.com/paper/2503.07591/) (2025)
- [Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learning](https://mlsanity.com/paper/2310.08782/) (2023)
- [Towards Free Data Selection with General-Purpose Models](https://mlsanity.com/paper/2309.17342/) (2023)
- [BAL: Balancing Diversity and Novelty for Active Learning](https://mlsanity.com/paper/2312.15944/) (2023)
- [Rethinking Epistemic and Aleatoric Uncertainty for Active Open-Set Annotation: An Energy-Based Approach](https://mlsanity.com/paper/2502.19691/) (2025)
- [Feature Alignment: Rethinking Efficient Active Learning via Proxy in the Context of Pre-trained Models](https://mlsanity.com/paper/2403.01101/) (2024)
- [NTKCPL: Active Learning on Top of Self-Supervised Model by Estimating True Coverage](https://mlsanity.com/paper/2306.04099/) (2023)
- [ActiveDC: Distribution Calibration for Active Finetuning](https://mlsanity.com/paper/2311.07634/) (2023)
- [Certainty and Uncertainty Guided Active Domain Adaptation](https://mlsanity.com/paper/2505.19421/) (2025)
- [VeCAF: Vision-language Collaborative Active Finetuning with Training Objective Awareness](https://mlsanity.com/paper/2401.07853/) (2024)
- [DCoM: Active Learning for All Learners](https://mlsanity.com/paper/2407.01804/) (2024)
- [Enhancing Semi-Supervised Learning via Representative and Diverse Sample Selection](https://mlsanity.com/paper/2409.11653/) (2024)
- [VIOLA: Towards Video In-Context Learning with Minimal Annotations](https://mlsanity.com/paper/2601.15549/) (2026)
- [Labeled TrustSet Guided: Batch Active Learning with Reinforcement Learning](https://mlsanity.com/paper/2604.12303/) (2026)
- [Enhancing Modality Representation and Alignment for Multimodal Cold-start Active Learning](https://mlsanity.com/paper/2412.09126/) (2024)
- [Efficient Code LLM Training via Distribution-Consistent and Diversity-Aware Data Selection](https://mlsanity.com/paper/2507.02378/) (2025)
- [Revisiting Unknowns: Towards Effective and Efficient Open-Set Active Learning](https://mlsanity.com/paper/2603.07898/) (2026)
- [Active Vision Reinforcement Learning under Limited Visual Observability](https://mlsanity.com/paper/2306.00975/) (2023)
- [Towards Clinical Practice in CT-Based Pulmonary Disease Screening: An Efficient and Reliable Framework](https://mlsanity.com/paper/2412.01525/) (2024)
- [Active Learning from Scene Embeddings for End-to-End Autonomous Driving](https://mlsanity.com/paper/2503.11062/) (2025)
- [Optimizing Active Learning in Vision-Language Models via Parameter-Efficient Uncertainty Calibration](https://mlsanity.com/paper/2507.21521/) (2025)

## Abstract (arXiv)

Given the large-scale data and the high annotation cost, pretraining-finetuning becomes a popular paradigm in multiple computer vision tasks. Previous research has covered both the unsupervised pretraining and supervised finetuning in this paradigm, while little attention is paid to exploiting the annotation budget for finetuning. To fill in this gap, we formally define this new active finetuning task focusing on the selection of samples for annotation in the pretraining-finetuning paradigm. We propose a novel method called ActiveFT for active finetuning task to select a subset of data distributing similarly with the entire unlabeled pool and maintaining enough diversity by optimizing a parametric model in the continuous space. We prove that the Earth Mover's distance between the distributions of the selected subset and the entire data pool is also reduced in this process. Extensive experiments show the leading performance and high efficiency of ActiveFT superior to baselines on both image classification and semantic segmentation. Our code is released at https://github.com/yichen928/ActiveFT.