# HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation

arXiv: https://arxiv.org/abs/2012.11582 · HTML: https://mlsanity.com/paper/2012.11582/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2020.

## Built on by (most cited first, 30 of 66)

- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [SegDiff: Image Segmentation with Diffusion Probabilistic Models](https://mlsanity.com/paper/2112.00390/) (2021)
- [HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing](https://mlsanity.com/paper/2111.15666/) (2021)
- [Highly Accurate Dichotomous Image Segmentation](https://mlsanity.com/paper/2203.03041/) (2022)
- [A Brief Review of Hypernetworks in Deep Learning](https://mlsanity.com/paper/2306.06955/) (2023)
- [HyperInverter: Improving StyleGAN Inversion via Hypernetwork](https://mlsanity.com/paper/2112.00719/) (2021)
- [PIDNet: A Real-time Semantic Segmentation Network Inspired by PID Controllers](https://mlsanity.com/paper/2206.02066/) (2022)
- [What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?](https://mlsanity.com/paper/2403.06090/) (2024)
- [Representation Compensation Networks for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.05402/) (2022)
- [Wireless Deep Video Semantic Transmission](https://mlsanity.com/paper/2205.13129/) (2022)
- [HyperE2VID: Improving Event-Based Video Reconstruction via Hypernetworks](https://mlsanity.com/paper/2305.06382/) (2023)
- [Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations](https://mlsanity.com/paper/2308.11796/) (2023)
- [Frequency-Adaptive Dilated Convolution for Semantic Segmentation](https://mlsanity.com/paper/2403.05369/) (2024)
- [Learning Local and Global Temporal Contexts for Video Semantic Segmentation](https://mlsanity.com/paper/2204.03330/) (2022)
- [DynaGAN: Dynamic Few-shot Adaptation of GANs to Multiple Domains](https://mlsanity.com/paper/2211.14554/) (2022)
- [HyperDet3D: Learning a Scene-conditioned 3D Object Detector](https://mlsanity.com/paper/2204.05599/) (2022)
- [Turbulence in Focus: Benchmarking Scaling Behavior of 3D Volumetric Super-Resolution with BLASTNet 2.0 Data](https://mlsanity.com/paper/2309.13457/) (2023)
- [HAIR: Hypernetworks-based All-in-One Image Restoration](https://mlsanity.com/paper/2408.08091/) (2024)
- [Rethinking Dilated Convolution for Real-time Semantic Segmentation](https://mlsanity.com/paper/2111.09957/) (2021)
- [Hyper-Convolution Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/2105.10559/) (2021)
- [MFNet: Multi-class Few-shot Segmentation Network with Pixel-wise Metric Learning](https://mlsanity.com/paper/2111.00232/) (2021)
- [SFNet: Faster and Accurate Semantic Segmentation via Semantic Flow](https://mlsanity.com/paper/2207.04415/) (2022)
- [Efficient Semantic Segmentation by Altering Resolutions for Compressed Videos](https://mlsanity.com/paper/2303.07224/) (2023)
- [Semantic Scene Segmentation for Robotics Applications](https://mlsanity.com/paper/2108.11128/) (2021)
- [On Efficient Real-Time Semantic Segmentation: A Survey](https://mlsanity.com/paper/2206.08605/) (2022)
- [Deep Learning based Computer Vision Methods for Complex Traffic Environments Perception: A Review](https://mlsanity.com/paper/2211.05120/) (2022)
- [Generalizable Neural Fields as Partially Observed Neural Processes](https://mlsanity.com/paper/2309.06660/) (2023)
- [Leveraging Human Selective Attention for Medical Image Analysis with Limited Training Data](https://mlsanity.com/paper/2112.01034/) (2021)
- [Hypernet-Ensemble Learning of Segmentation Probability for Medical Image Segmentation with Ambiguous Labels](https://mlsanity.com/paper/2112.06693/) (2021)
- [Lightweight and Progressively-Scalable Networks for Semantic Segmentation](https://mlsanity.com/paper/2207.13600/) (2022)

## Abstract (arXiv)

We present a novel, real-time, semantic segmentation network in which the encoder both encodes and generates the parameters (weights) of the decoder. Furthermore, to allow maximal adaptivity, the weights at each decoder block vary spatially. For this purpose, we design a new type of hypernetwork, composed of a nested U-Net for drawing higher level context features, a multi-headed weight generating module which generates the weights of each block in the decoder immediately before they are consumed, for efficient memory utilization, and a primary network that is composed of novel dynamic patch-wise convolutions. Despite the usage of less-conventional blocks, our architecture obtains real-time performance. In terms of the runtime vs. accuracy trade-off, we surpass state of the art (SotA) results on popular semantic segmentation benchmarks: PASCAL VOC 2012 (val. set) and real-time semantic segmentation on Cityscapes, and CamVid. The code is available: https://nirkin.com/hyperseg.