mlsanityData to 7 Oct 2026

FACT: A Simple and Efficient Framework for Active Finetuning

arXiv 2606.02079 ↗First posted 1 Jun 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

The main goal of active finetuning is to improve a pretrained model's performance on a specific task or domain by finetuning it with carefully selected informative or challenging data. Previous research has predominantly focused on the active aspect (i.e., data selection) while uniformly employing full finetuning for model adaptation, which inevitably distorts pretrained features due to distribution shift. This issue becomes particularly pronounced when the model size is large relative to the finetuning data quantity, leading to heightened overfitting risks. To address this critical gap, we formally outline the FiAF task that emphasizes systematic exploration of finetuning methodologies in active learning. We propose FACT, a three-phase hierarchical finetuning framework featuring both efficiency and simplicity, specifically designed for active finetuning scenarios. Our comprehensive experiments span: (1) Three major dataset categories encompassing classic (CIFAR10, CIFAR100, ImageNet-1k), imbalanced (CIFAR10-LT, CIFAR100-LT), and fine-grained (StanfordCars, FGVCAircraft) image classification datasets, each evaluated under 3-5 distinct sampling ratios; (2) Diverse pretrained architectures including Convolutional Neural Network (ConvNeXt), Vision Transformer (ViT), and Vision LSTM (ViL) networks; (3) A systematic investigation of frozen feature augmentation (FroFA) strategies. (4) A comprehensive and rigorous analysis of efficiency and generalizability. The results demonstrate significant improvements with strong generalization and robustness. Notably, under low sampling ratios, our framework achieves remarkable performance gains of over 20% on the ViT model for CIFAR10, CIFAR100, and ImageNet-1k benchmarks. This systematic approach establishes new state-of-the-art performance while maintaining parameter efficiency, proving particularly effective when labeled data is scarce.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 58 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

43 of 58 references.

References · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2022A ConvNet for the 2020scited by 3,545
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  12. 2022Visual Prompt Tuningcited by 1,674
31 more
  1. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  2. 2021Scaling Vision Transformerscited by 993
  3. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  4. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  5. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  6. 2019A Closer Look at Few-shot Classificationcited by 679
  7. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  8. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  9. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  10. 2019Learning Loss for Active Learningcited by 335
  11. 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 225
  12. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  13. 2019Charting the Right Manifold: Manifold Mixup for Few-shot Learningcited by 135
  14. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  15. 2022Active Learning on a Budget: Opposite Strategies Suit High and Low Budgetscited by 131
  16. 2022Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Differencecited by 123
  17. 2020Task-Aware Variational Adversarial Active Learningcited by 84
  18. 2021Partial Is Better Than All: Revisiting Fine-tuning Strategy for Few-shot Learningcited by 72
  19. 2022Active Learning by Feature Mixingcited by 63
  20. 2024Vision-LSTM: xLSTM as Generic Vision Backbonecited by 51
  21. 2018A Progressive Batching L-BFGS Method for Machine Learningcited by 45
  22. 2024Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learningcited by 43
  23. 2021Reducing Label Effort: Self-Supervised meets Active Learningcited by 42
  24. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  25. 2023Active Finetuning: Exploiting Annotation Budget in the Pretraining-Finetuning Paradigmcited by 36
  26. 2022Equi-Tuning: Group Equivariant Fine-Tuning of Pretrained Modelscited by 22
  27. 2022Deep Active Learning with Noise Stabilitycited by 13
  28. 2024Frozen Feature Augmentation for Few-Shot Image Classificationcited by 13
  29. 2020CODE: Contrastive Pre-training with Adversarial Fine-tuning for Zero-shot Expert Linkingcited by 3
  30. 2023ActiveDC: Distribution Calibration for Active Finetuningcited by 3
  31. 2024VeCAF: Vision-language Collaborative Active Finetuning with Training Objective Awarenesscited by 2

Built on by

Not cited yet.