mlsanityData to 14 Oct 2026

FisherTune: Fisher-Guided Robust Tuning of Vision Foundation Models for Domain Generalized Segmentation

arXiv 2503.17940 ↗First posted 23 Mar 2025cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

Vision Foundation Models (VFMs) excel in generalization due to large-scale pretraining, but fine-tuning them for Domain Generalized Semantic Segmentation (DGSS) while maintaining this ability remains challenging. Existing approaches either selectively fine-tune parameters or freeze the VFMs and update only the adapters, both of which may underutilize the VFMs' full potential in DGSS tasks. We observe that domain-sensitive parameters in VFMs, arising from task and distribution differences, can hinder generalization. To address this, we propose FisherTune, a robust fine-tuning method guided by the Domain-Related Fisher Information Matrix (DR-FIM). DR-FIM measures parameter sensitivity across tasks and domains, enabling selective updates that preserve generalization and enhance DGSS adaptability. FisherTune incorporates variational inference to stabilize DR-FIM estimation, treating parameters as Gaussian-distributed variables and leveraging pre-trained priors. Extensive experiments show that FisherTune achieves superior cross-domain segmentation while maintaining generalization, outperforming selective-parameter and adapter-based methods.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 64 of its 84 references that are in the map. First posted on arXiv 23 Mar 2025.

Builds on

64 of 84 references.

References · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Segment Anythingcited by 8,234
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  9. 2022Visual Prompt Tuningcited by 1,681
  10. 2016Variational Inference: A Review for Statisticianscited by 1,472
  11. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  12. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
38 more
  1. 2016Playing for Data: Ground Truth from Computer Gamescited by 818
  2. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 763
  3. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  4. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  5. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  6. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  7. 2015Variational Dropout and the Local Reparameterization Trickcited by 490
  8. 2019Domain Generalization via Model-Agnostic Learning of Semantic Featurescited by 407
  9. 2017Semantic Foggy Scene Understanding with Synthetic Datacited by 403
  10. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 382
  11. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 308
  12. 2019Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Datacited by 246
  13. 2021RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whiteningcited by 215
  14. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  15. 2019Task2Vec: Task Embedding for Meta-Learningcited by 205
  16. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  17. 2017Image to Image Translation for Domain Adaptationcited by 171
  18. 2021FSDR: Frequency Space Domain Randomization for Domain Generalizationcited by 164
  19. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  20. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 146
  21. 2022Uncertainty Modeling for Out-of-Distribution Generalizationcited by 125
  22. 2018Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttimecited by 122
  23. 2023Explicit Visual Prompting for Low-Level Structure Segmentationscited by 114
  24. 2018Model Adaptation with Synthetic and Real Data for Semantic Dense Foggy Scene Understandingcited by 98
  25. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  26. 2023Dynamic Conceptional Contrastive Learning for Generalized Category Discoverycited by 92
  27. 2022Style-Hallucinated Dual Consistency Learning for Domain Generalized Semantic Segmentationcited by 89
  28. 2023Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentationcited by 83
  29. 2022Semantic-Aware Domain Generalized Segmentationcited by 79
  30. 2022WildNet: Learning Domain Generalized Semantic Segmentation from the Wildcited by 76
  31. 2022Adversarial Style Augmentation for Domain Generalized Urban-Scene Segmentationcited by 70
  32. 2022PODA: Prompt-driven Zero-shot Domain Adaptationcited by 60
  33. 2023Parameter-Efficient Fine-Tuning without Introducing New Latencycited by 59
  34. 2021Global and Local Texture Randomization for Synthetic-to-Real Semantic Segmentationcited by 57
  35. 2022Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learningcited by 57
  36. 2023Source-Free Domain Adaptation with Frozen Multimodal Foundation Modelcited by 55
  37. 2019Where is the Information in a Deep Neural Network?cited by 42
  38. 2022PASTA: Proportional Amplitude Spectrum Training Augmentation for Syn-to-Real Domain Generalizationcited by 39

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 9.

16 papers.

Sort:

By area:

  1. 2025OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interactioncited by 14
  2. 2025Exploring Probabilistic Modeling Beyond Domain Generalization for Semantic Segmentationcited by 4
  3. 2026Open-Vocabulary Domain Generalization in Urban-Scene Segmentationcited by 4
  4. 2025Rein++: Efficient Generalization and Adaptation for Semantic Segmentation with Vision Foundation Modelscited by 2
  5. 2025Causal-Tune: Mining Causal Factors from Vision Foundation Models for Domain Generalized Semantic Segmentationcited by 2
  6. 2025Orthogonal Projection Subspace to Aggregate Online Prior-knowledge for Continual Test-time Adaptationcited by 1
  7. 2026CLAP: Contrastive Latent-space Prompt Optimization for End-to-end Autonomous Drivingcited by 1
  8. 2025Multi-Granularity Feature Calibration via VFM for Domain Generalized Semantic Segmentationcited by 0
  9. 2025Domain Generalization for Semantic Segmentation: A Surveycited by 0
  10. 2026DSP-Reg: Domain-Sensitive Parameter Regularization for Robust Domain Generalizationcited by 0
  11. 2026Generalizable Knowledge Distillation from Vision Foundation Models for Semantic Segmentationcited by 0
  12. 2026Fisher-Guided Progressive Parameter Selection for Adaptive Fine-Tuningcited by 0
4 more
  1. 2026LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuningcited by 0
  2. 2026Ultra: Unsupervised Cross-Task Optimization for Reliable Restoration Segmentation Collaboration under Adverse Weathercited by 0
  3. 2026Hierarchical Prompt Injector for Domain Generalization Segmentationcited by 0
  4. 2026Sparse All-Layer Connector for Domain Generalised Semantic Segmentationcited by 0