# Fast, Exact and Multi-Scale Inference for Semantic Image Segmentation with Deep Gaussian CRFs

arXiv: https://arxiv.org/abs/1603.08358 · HTML: https://mlsanity.com/paper/1603.08358/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2016.

## Built on by (most cited first, 27 of 27)

- [Pseudo-mask Matters in Weakly-supervised Semantic Segmentation](https://mlsanity.com/paper/2108.12995/) (2021)
- [Look Before You Match: Instance Understanding Matters in Video Object Segmentation](https://mlsanity.com/paper/2212.06826/) (2022)
- [ProtoSeg: Interpretable Semantic Segmentation with Prototypical Parts](https://mlsanity.com/paper/2301.12276/) (2023)
- [Label Propagation for Zero-shot Classification with Vision-Language Models](https://mlsanity.com/paper/2404.04072/) (2024)
- [Semantic Image Segmentation: Two Decades of Research](https://mlsanity.com/paper/2302.06378/) (2023)
- [On the Road to Online Adaptation for Semantic Image Segmentation](https://mlsanity.com/paper/2203.16195/) (2022)
- [LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentation](https://mlsanity.com/paper/2503.19777/) (2025)
- [Unsupervised Domain Adaptation for Semantic Image Segmentation: a Comprehensive Survey](https://mlsanity.com/paper/2112.03241/) (2021)
- [Regularized Frank-Wolfe for Dense CRFs: Generalizing Mean Field and Beyond](https://mlsanity.com/paper/2110.14759/) (2021)
- [Boosting Night-time Scene Parsing with Learnable Frequency](https://mlsanity.com/paper/2208.14241/) (2022)
- [Learning Structured Gaussians to Approximate Deep Ensembles](https://mlsanity.com/paper/2203.15485/) (2022)
- [Semantic Segmentation using Neural Ordinary Differential Equations](https://mlsanity.com/paper/2209.08667/) (2022)
- [Modeling Label Correlations for Ultra-Fine Entity Typing with Neural Pairwise Conditional Random Field](https://mlsanity.com/paper/2212.01581/) (2022)
- [STAT: Towards Generalizable Temporal Action Localization](https://mlsanity.com/paper/2404.13311/) (2024)
- [MoE-SPNet: A Mixture-of-Experts Scene Parsing Network](https://mlsanity.com/paper/1806.07049/) (2018)
- [DeVOS: Flow-Guided Deformable Transformer for Video Object Segmentation](https://mlsanity.com/paper/2405.08715/) (2024)
- [ROGPL: Robust Open-Set Graph Learning via Region-Based Prototype Learning](https://mlsanity.com/paper/2402.18495/) (2024)
- [Border-SegGCN: Improving Semantic Segmentation by Refining the Border Outline using Graph Convolutional Network](https://mlsanity.com/paper/2109.05353/) (2021)
- [Denoised Non-Local Neural Network for Semantic Segmentation](https://mlsanity.com/paper/2110.14200/) (2021)
- [On the Effect of Image Resolution on Semantic Segmentation](https://mlsanity.com/paper/2402.05398/) (2024)
- [Global Motion Understanding in Large-Scale Video Object Segmentation](https://mlsanity.com/paper/2405.07031/) (2024)
- [Use square root affinity to regress labels in semantic segmentation](https://mlsanity.com/paper/2103.04990/) (2021)
- [Multi-Scale Semantic Segmentation with Modified MBConv Blocks](https://mlsanity.com/paper/2402.04618/) (2024)
- [Early Fusion of Features for Semantic Segmentation](https://mlsanity.com/paper/2402.06091/) (2024)
- [SGD with Partial Hessian for Deep Neural Networks Optimization](https://mlsanity.com/paper/2403.02681/) (2024)
- [Your Image is My Video: Reshaping the Receptive Field via Image-To-Video Differentiable AutoAugmentation and Fusion](https://mlsanity.com/paper/2403.15194/) (2024)
- [UnfoldCRF: Structured Mask Refinement with Image-Conditioned Latent Regions](https://mlsanity.com/paper/2609.33996/) (2026)

## Abstract (arXiv)

In this work we propose a structured prediction technique that combines the virtues of Gaussian Conditional Random Fields (G-CRF) with Deep Learning: (a) our structured prediction task has a unique global optimum that is obtained exactly from the solution of a linear system (b) the gradients of our model parameters are analytically computed using closed form expressions, in contrast to the memory-demanding contemporary deep structured prediction approaches that rely on back-propagation-through-time, (c) our pairwise terms do not have to be simple hand-crafted expressions, as in the line of works building on the DenseCRF, but can rather be `discovered' from data through deep architectures, and (d) out system can trained in an end-to-end manner. Building on standard tools from numerical analysis we develop very efficient algorithms for inference and learning, as well as a customized technique adapted to the semantic segmentation task. This efficiency allows us to explore more sophisticated architectures for structured prediction in deep learning: we introduce multi-resolution architectures to couple information across scales in a joint optimization framework, yielding systematic improvements. We demonstrate the utility of our approach on the challenging VOC PASCAL 2012 image segmentation benchmark, showing substantial improvements over strong baselines. We make all of our code and experiments available at {https://github.com/siddharthachandra/gcrf}