# SegDem: Segmentation helps Demosaicing

arXiv: https://arxiv.org/abs/2608.07916 · HTML: https://mlsanity.com/paper/2608.07916/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 46 references that are in the map.
- First posted on arXiv 8 Aug 2026.

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space](https://mlsanity.com/paper/2506.15742/) (2025)
- [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [Exploiting Diffusion Prior for Real-World Image Super-Resolution](https://mlsanity.com/paper/2305.07015/) (2023)
- [MambaIR: A Simple Baseline for Image Restoration with State-Space Model](https://mlsanity.com/paper/2402.15648/) (2024)
- [SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution](https://mlsanity.com/paper/2311.16518/) (2023)
- [DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior](https://mlsanity.com/paper/2308.15070/) (2023)
- [Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization](https://mlsanity.com/paper/2308.14469/) (2023)
- [One-Step Effective Diffusion Network for Real-World Image Super-Resolution](https://mlsanity.com/paper/2406.08177/) (2024)
- [Controlling Vision-Language Models for Multi-Task Image Restoration](https://mlsanity.com/paper/2310.01018/) (2023)
- [What matters for Representation Alignment: Global Information or Spatial Structure?](https://mlsanity.com/paper/2512.10794/) (2025)
- [Vision-Language Gradient Descent-driven All-in-One Deep Unfolding Networks](https://mlsanity.com/paper/2503.16930/) (2025)
- [Deep Image Demosaicking using a Cascade of Convolutional Residual Denoising Networks](https://mlsanity.com/paper/1803.05215/) (2018)
- [Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models](https://mlsanity.com/paper/2605.06388/) (2026)
- [Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning](https://mlsanity.com/paper/2410.19560/) (2024)
- [Consistency Trajectory Matching for One-Step Generative Super-Resolution](https://mlsanity.com/paper/2503.20349/) (2025)
- [RealRestorer: Towards Generalizable Real-World Image Restoration with Large-Scale Image Editing Models](https://mlsanity.com/paper/2603.25502/) (2026)
- [MambaSCI: Efficient Mamba-UNet for Quad-Bayer Patterned Video Snapshot Compressive Imaging](https://mlsanity.com/paper/2410.14214/) (2024)
- [Binarized Mamba-Transformer for Lightweight Quad Bayer HybridEVS Demosaicing](https://mlsanity.com/paper/2503.16134/) (2025)
- [Efficient Unified Demosaicing for Bayer and Non-Bayer Patterned Image Sensors](https://mlsanity.com/paper/2307.10667/) (2023)
- [Inheriting Bayer's Legacy-Joint Remosaicing and Denoising for Quad Bayer Image Sensor](https://mlsanity.com/paper/2303.13571/) (2023)
- [Can Nano Banana 2 Replace Traditional Image Restoration Models? An Evaluation of Its Performance on Image Restoration Tasks](https://mlsanity.com/paper/2604.03061/) (2026)
- [Examining Joint Demosaicing and Denoising for Single-, Quad-, and Nona-Bayer Patterns](https://mlsanity.com/paper/2504.07145/) (2025)

## Abstract (arXiv)

Image demosaicing reconstructs a full-color image from incomplete color measurements produced by a sensor covered with a color filter array (CFA). Most existing methods formulate demosaicing as pixel-level reconstruction and mainly rely on local textures, cross-channel correlations, and low-level image statistics. Our core insight is that reconstruction and visual understanding can be viewed as complementary views of shared scene structure: both are grounded in the same underlying physical world, and therefore the structural and physical information inferred from an image should remain consistent across the two tasks. We instantiate this idea with instance segmentation and propose SegDem, a cross-task decoder representation transfer framework for demosaicing. SegDem first learns region- and boundary-aware representations through instance-aware structural pretraining and then transfers the decoder to RAW-conditioned reconstruction. Segmentation- and demosaicing-conditioned features are further anchored to a shared frozen DINOv2 representation space to preserve structural organization across tasks. We instantiate SegDem with convolutional, Transformer-based, and state-space backbones for unified Single- and Quad-Bayer demosaicing. Extensive experiments on synthetic, external, and challenging datasets demonstrate consistent improvements across different architectures and CFA layouts.