Unleashing Degradation-Carrying Features in Symmetric U-Net: Simpler and Stronger Baselines for All-in-One Image Restoration

Current methods for all-in-one image restoration increasingly rely on complex mechanisms, such as Mixture-of-Experts or prompt learning, to handle diverse degradations.In this work, we challenge this trend by showing that structural symmetry provides a strong and effective inductive bias, which is often sufficient to support high-quality restoration in many practical settings. We propose SymUNet, a minimalist baseline that enforces strict encoder-decoder alignment via simple additive fusion, which outperforms complex state-of-the-art models on standard benchmarks while significantly reducing computational costs. Furthermore, to address the negative transfer observed in highly diverse degradation scenarios (e.g., 5-task settings), we introduce SE-SymUNet, which leverages frozen semantic priors (CLIP) to disentangle conflicting restoration tasks. Extensive experiments demonstrate that our approach establishes a simpler, stronger, and more efficient foundation for multi-degradation learning.