# Learning Spectral-Decomposed Tokens for Domain Generalized Semantic Segmentation

arXiv: https://arxiv.org/abs/2407.18568 · HTML: https://mlsanity.com/paper/2407.18568/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 69 references that are in the map.
- First posted on arXiv 26 Jul 2024.

## Builds on (most cited first, 30 of 44)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [FDA: Fourier Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2004.05498/) (2020)
- [Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net](https://mlsanity.com/paper/1807.09441/) (2018)
- [A Fourier-based Framework for Domain Generalization](https://mlsanity.com/paper/2105.11120/) (2021)
- [Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Data](https://mlsanity.com/paper/1909.00889/) (2019)
- [RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening](https://mlsanity.com/paper/2103.15597/) (2021)
- [SelfReg: Self-supervised Contrastive Regularization for Domain Generalization](https://mlsanity.com/paper/2104.09841/) (2021)
- [FSDR: Frequency Space Domain Randomization for Domain Generalization](https://mlsanity.com/paper/2103.02370/) (2021)
- [Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications](https://mlsanity.com/paper/2304.05750/) (2023)
- [Switchable Whitening for Deep Representation Learning](https://mlsanity.com/paper/1904.09739/) (2019)
- [Iterative Normalization: Beyond Standardization towards Efficient Whitening](https://mlsanity.com/paper/1904.03441/) (2019)
- [Style-Hallucinated Dual Consistency Learning for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2204.02548/) (2022)
- [Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2312.04265/) (2023)
- [Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks](https://mlsanity.com/paper/1805.07925/) (2018)
- [Meta Batch-Instance Normalization for Generalizable Person Re-Identification](https://mlsanity.com/paper/2011.14670/) (2020)
- [WildNet: Learning Domain Generalized Semantic Segmentation from the Wild](https://mlsanity.com/paper/2204.01446/) (2022)
- [Amplitude-Phase Recombination: Rethinking Robustness of Convolutional Neural Networks in Frequency Domain](https://mlsanity.com/paper/2108.08487/) (2021)
- [Adversarial Style Augmentation for Domain Generalized Urban-Scene Segmentation](https://mlsanity.com/paper/2207.04892/) (2022)
- [Adversarial Style Mining for One-Shot Unsupervised Domain Adaptation](https://mlsanity.com/paper/2004.06042/) (2020)
- [Batch Normalization Embeddings for Deep Domain Generalization](https://mlsanity.com/paper/2011.12672/) (2020)
- [Global and Local Texture Randomization for Synthetic-to-Real Semantic Segmentation](https://mlsanity.com/paper/2108.02376/) (2021)

## Built on by (most cited first, 22 of 22)

- [SoMA: Singular Value Decomposed Minor Components Adaptation for Domain Generalizable Representation Learning](https://mlsanity.com/paper/2412.04077/) (2024)
- [Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language Models for Domain-Generalized Semantic Segmentation](https://mlsanity.com/paper/2504.03193/) (2025)
- [FisherTune: Fisher-Guided Robust Tuning of Vision Foundation Models for Domain Generalized Segmentation](https://mlsanity.com/paper/2503.17940/) (2025)
- [Multi-source Domain Adaptation for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2408.16469/) (2024)
- [SSH: Sparse Spectrum Adaptation via Discrete Hartley Transformation](https://mlsanity.com/paper/2502.05539/) (2025)
- [Learning Fine-grained Domain Generalization via Hyperbolic State Space Hallucination](https://mlsanity.com/paper/2504.08020/) (2025)
- [BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2410.09758/) (2024)
- [ER-LoRA: Effective-Rank Guided Adaptation for Weather-Generalized Depth Estimation](https://mlsanity.com/paper/2509.00665/) (2025)
- [Open-Vocabulary Domain Generalization in Urban-Scene Segmentation](https://mlsanity.com/paper/2602.18853/) (2026)
- [Generalizable Multispectral Land Cover Classification via Frequency-Aware Mixture of Low-Rank Token Experts](https://mlsanity.com/paper/2505.14088/) (2025)
- [Diversity Covariance-Aware Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2503.01531/) (2025)
- [Rein++: Efficient Generalization and Adaptation for Semantic Segmentation with Vision Foundation Models](https://mlsanity.com/paper/2508.01667/) (2025)
- [Causal-Tune: Mining Causal Factors from Vision Foundation Models for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2512.16567/) (2025)
- [DGFamba: Learning Flow Factorized State Space for Visual Domain Generalization](https://mlsanity.com/paper/2504.08019/) (2025)
- [SS-DC: Spatial-Spectral Decoupling and Coupling Across Visible-Infrared Gap for Domain Adaptive Object Detection](https://mlsanity.com/paper/2507.12017/) (2025)
- [Local Precise Refinement: A Dual-Gated Mixture-of-Experts for Enhancing Foundation Model Generalization against Spectral Shifts](https://mlsanity.com/paper/2603.13352/) (2026)
- [Multi-Granularity Feature Calibration via VFM for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2508.03007/) (2025)
- [Domain Generalization for Semantic Segmentation: A Survey](https://mlsanity.com/paper/2510.03540/) (2025)
- [SAGE: Style-Adaptive Generalization for Privacy-Constrained Semantic Segmentation Across Domains](https://mlsanity.com/paper/2512.02369/) (2025)
- [RecycleLoRA: Rank-Revealing QR-Based Dual-LoRA Subspace Adaptation for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2603.28142/) (2026)
- [Hierarchical Prompt Injector for Domain Generalization Segmentation](https://mlsanity.com/paper/2609.05864/) (2026)
- [Sparse All-Layer Connector for Domain Generalised Semantic Segmentation](https://mlsanity.com/paper/2610.15660/) (2026)

## Abstract (arXiv)

The rapid development of Vision Foundation Model (VFM) brings inherent out-domain generalization for a variety of down-stream tasks. Among them, domain generalized semantic segmentation (DGSS) holds unique challenges as the cross-domain images share common pixel-wise content information but vary greatly in terms of the style. In this paper, we present a novel Spectral-dEcomposed Token (SET) learning framework to advance the frontier. Delving into further than existing fine-tuning token & frozen backbone paradigm, the proposed SET especially focuses on the way learning style-invariant features from these learnable tokens. Particularly, the frozen VFM features are first decomposed into the phase and amplitude components in the frequency space, which mainly contain the information of content and style, respectively, and then separately processed by learnable tokens for task-specific information extraction. After the decomposition, style variation primarily impacts the token-based feature enhancement within the amplitude branch. To address this issue, we further develop an attention optimization method to bridge the gap between style-affected representation and static tokens during inference. Extensive cross-domain experiments show its state-of-the-art performance.