# Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learning

arXiv: https://arxiv.org/abs/2403.00567 · HTML: https://mlsanity.com/paper/2403.00567/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 1 Mar 2024.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [Using Deep Learning for Image-Based Plant Disease Detection](https://mlsanity.com/paper/1604.03169/) (2016)
- [Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation](https://mlsanity.com/paper/2001.08735/) (2020)
- [Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference](https://mlsanity.com/paper/2204.07305/) (2022)
- [Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima](https://mlsanity.com/paper/2111.01549/) (2021)
- [Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks](https://mlsanity.com/paper/1805.07925/) (2018)
- [Meta Batch-Instance Normalization for Generalizable Person Re-Identification](https://mlsanity.com/paper/2011.14670/) (2020)
- [Cross-Domain Few-Shot Classification via Adversarial Task Augmentation](https://mlsanity.com/paper/2104.14385/) (2021)
- [Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization](https://mlsanity.com/paper/2303.03108/) (2023)
- [TaskNorm: Rethinking Batch Normalization for Meta-Learning](https://mlsanity.com/paper/2003.03284/) (2020)
- [Margin-Based Few-Shot Class-Incremental Learning with Class-Level Overfitting Mitigation](https://mlsanity.com/paper/2210.04524/) (2022)
- [Adversarial Feature Augmentation for Cross-domain Few-shot Classification](https://mlsanity.com/paper/2208.11021/) (2022)
- [Ranking Distance Calibration for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2112.00260/) (2021)
- [Wave-SAN: Wavelet based Style Augmentation Network for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2203.07656/) (2022)
- [On Separate Normalization in Self-supervised Transformers](https://mlsanity.com/paper/2309.12931/) (2023)

## Built on by (most cited first, 26 of 26)

- [The Devil is in Low-Level Features for Cross-Domain Few-Shot Segmentation](https://mlsanity.com/paper/2503.21150/) (2025)
- [Revisiting Pool-based Prompt Learning for Few-shot Class-incremental Learning](https://mlsanity.com/paper/2507.09183/) (2025)
- [Mamba-FSCIL: Dynamic Adaptation with Selective State Space Model for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2407.06136/) (2024)
- [Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2506.07376/) (2025)
- [VT-FSL: Bridging Vision and Text with LLMs for Few-Shot Learning](https://mlsanity.com/paper/2509.25033/) (2025)
- [Reclaiming Lost Text Layers for Source-Free Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2603.05235/) (2026)
- [Random Registers for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2506.02843/) (2025)
- [Mind the Discriminability Trap in Source-Free Cross-domain Few-shot Learning](https://mlsanity.com/paper/2603.13341/) (2026)
- [Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignment](https://mlsanity.com/paper/2603.17655/) (2026)
- [Revisiting Continuity of Image Tokens for Cross-domain Few-shot Learning](https://mlsanity.com/paper/2506.03110/) (2025)
- [DVLA-RL: Dual-Level Vision-Language Alignment with Reinforcement Learning Gating for Few-Shot Learning](https://mlsanity.com/paper/2602.00795/) (2026)
- [Remedying Target-Domain Astigmatism for Cross-Domain Few-Shot Object Detection](https://mlsanity.com/paper/2603.18541/) (2026)
- [Addressing Exacerbated Attention Sink for Source-Free Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2605.25799/) (2026)
- [Probability Distribution Learning and Its Application in Deep Learning](https://mlsanity.com/paper/2406.05666/) (2024)
- [Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2605.29776/) (2026)
- [MetaLab: Few-Shot Game Changer for Image Recognition](https://mlsanity.com/paper/2507.22057/) (2025)
- [Color as the Impetus: Transforming Few-Shot Learner](https://mlsanity.com/paper/2507.22136/) (2025)
- [FreqGRL: Suppressing Low-Frequency Bias and Mining High-Frequency Knowledge for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2511.06648/) (2025)
- [IPEC: Test-Time Incremental Prototype Enhancement Classifier for Few-Shot Learning](https://mlsanity.com/paper/2601.11669/) (2026)
- [PMCE: Probabilistic Multi-Granularity Semantics with Caption-Guided Enhancement for Few-Shot Learning](https://mlsanity.com/paper/2601.14111/) (2026)
- [SRasP: Self-Reorientation Adversarial Style Perturbation for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2603.05135/) (2026)
- [GiPL: Generative augmented iterative Pseudo-Labeling for Cross-Domain Few-Shot Object Detection](https://mlsanity.com/paper/2605.29539/) (2026)
- [Free-Lunch Augmentation by Revisiting Diffusion-Based Data Generation for Cross-Domain Few-Shot Object Detection](https://mlsanity.com/paper/2608.04394/) (2026)
- [The First EgoCross Challenge at EgoVis 2026: Cross-Domain Egocentric Video Question Answering](https://mlsanity.com/paper/2608.04589/) (2026)
- [When Semantics Saturate or Emerge: Adaptation-Conditional Semantic Utility in Source-Free Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2608.06673/) (2026)
- [Training-Free Spectral Transductive Refinement for Cross-Domain Few-Shot Classification](https://mlsanity.com/paper/2609.23758/) (2026)

## Abstract (arXiv)

Cross-domain few-shot learning (CDFSL) aims to acquire knowledge from limited training data in the target domain by leveraging prior knowledge transferred from source domains with abundant training samples. CDFSL faces challenges in transferring knowledge across dissimilar domains and fine-tuning models with limited training data. To address these challenges, we initially extend the analysis of loss landscapes from the parameter space to the representation space, which allows us to simultaneously interpret the transferring and fine-tuning difficulties of CDFSL models. We observe that sharp minima in the loss landscapes of the representation space result in representations that are hard to transfer and fine-tune. Moreover, existing flatness-based methods have limited generalization ability due to their short-range flatness. To enhance the transferability and facilitate fine-tuning, we introduce a simple yet effective approach to achieve long-range flattening of the minima in the loss landscape. This approach considers representations that are differently normalized as minima in the loss landscape and flattens the high-loss region in the middle by randomly sampling interpolated representations. We implement this method as a new normalization layer that replaces the original one in both CNNs and ViTs. This layer is simple and lightweight, introducing only a minimal number of additional parameters. Experimental results on 8 datasets demonstrate that our approach outperforms state-of-the-art methods in terms of average accuracy. Moreover, our method achieves performance improvements of up to 9% compared to the current best approaches on individual datasets. Our code will be released.