# MSFA-Frequency-Aware Transformer for Hyperspectral Images Demosaicing

arXiv: https://arxiv.org/abs/2303.13404 · HTML: https://mlsanity.com/paper/2303.13404/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 55 references that are in the map.
- First posted on arXiv 23 Mar 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better](https://mlsanity.com/paper/1908.03826/) (2019)
- [Rethinking Coarse-to-Fine Approach in Single Image Deblurring](https://mlsanity.com/paper/2108.05054/) (2021)
- [HINet: Half Instance Normalization Network for Image Restoration](https://mlsanity.com/paper/2105.06086/) (2021)
- [Focal Frequency Loss for Image Reconstruction and Synthesis](https://mlsanity.com/paper/2012.12821/) (2020)
- [TransFG: A Transformer Architecture for Fine-grained Recognition](https://mlsanity.com/paper/2103.07976/) (2021)
- [Video Super-Resolution Transformer](https://mlsanity.com/paper/2106.06847/) (2021)
- [Rank Minimization for Snapshot Compressive Imaging](https://mlsanity.com/paper/1807.07837/) (2018)
- [Unsupervised Real-world Image Super Resolution via Domain-distance Aware Training](https://mlsanity.com/paper/2004.01178/) (2020)
- [Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging](https://mlsanity.com/paper/2205.10102/) (2022)
- [Spatial-Temporal Transformer for Video Snapshot Compressive Imaging](https://mlsanity.com/paper/2209.01578/) (2022)
- [Towards Interpretable Video Super-Resolution via Alternating Optimization](https://mlsanity.com/paper/2207.10765/) (2022)
- [Deep demosaicking for multispectral filter arrays](https://mlsanity.com/paper/1808.08021/) (2018)
- [Transformer-based normative modelling for anomaly detection of early schizophrenia](https://mlsanity.com/paper/2212.04984/) (2022)

## Built on by (most cited first, 1 of 1)

- [Event Camera Demosaicing via Swin Transformer and Pixel-focus Loss](https://mlsanity.com/paper/2404.02731/) (2024)

## Abstract (arXiv)

Hyperspectral imaging systems that use multispectral filter arrays (MSFA) capture only one spectral component in each pixel. Hyperspectral demosaicing is used to recover the non-measured components. While deep learning methods have shown promise in this area, they still suffer from several challenges, including limited modeling of non-local dependencies, lack of consideration of the periodic MSFA pattern that could be linked to periodic artifacts, and difficulty in recovering high-frequency details. To address these challenges, this paper proposes a novel de-mosaicing framework, the MSFA-frequency-aware Transformer network (FDM-Net). FDM-Net integrates a novel MSFA-frequency-aware multi-head self-attention mechanism (MaFormer) and a filter-based Fourier zero-padding method to reconstruct high pass components with greater difficulty and low pass components with relative ease, separately. The advantage of Maformer is that it can leverage the MSFA information and non-local dependencies present in the data. Additionally, we introduce a joint spatial and frequency loss to transfer MSFA information and enhance training on frequency components that are hard to recover. Our experimental results demonstrate that FDM-Net outperforms state-of-the-art methods with 6dB PSNR, and reconstructs high-fidelity details successfully.