# Higher Order Conditional Random Fields in Deep Neural Networks

arXiv: https://arxiv.org/abs/1511.08119 · HTML: https://mlsanity.com/paper/1511.08119/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2015.

## Built on by (most cited first, 26 of 26)

- [Representation Compensation Networks for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.05402/) (2022)
- [ReMaX: Relaxing for Better Training on Efficient Panoptic Segmentation](https://mlsanity.com/paper/2306.17319/) (2023)
- [Continual Learning for Image Segmentation with Dynamic Query](https://mlsanity.com/paper/2311.17450/) (2023)
- [Dynamic Neural Representational Decoders for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/2107.14428/) (2021)
- [Regularized Frank-Wolfe for Dense CRFs: Generalizing Mean Field and Beyond](https://mlsanity.com/paper/2110.14759/) (2021)
- [Probabilistic Graph Attention Network with Conditional Kernels for Pixel-Wise Prediction](https://mlsanity.com/paper/2101.02843/) (2021)
- [ORDNet: Capturing Omni-Range Dependencies for Scene Parsing](https://mlsanity.com/paper/2101.03929/) (2021)
- [E-CRF: Embedded Conditional Random Field for Boundary-caused Class Weights Confusion in Semantic Segmentation](https://mlsanity.com/paper/2112.07106/) (2021)
- [Modeling Label Correlations for Ultra-Fine Entity Typing with Neural Pairwise Conditional Random Field](https://mlsanity.com/paper/2212.01581/) (2022)
- [MoE-SPNet: A Mixture-of-Experts Scene Parsing Network](https://mlsanity.com/paper/1806.07049/) (2018)
- [DPF: Learning Dense Prediction Fields with Weak Supervision](https://mlsanity.com/paper/2303.16890/) (2023)
- [Variational Structured Attention Networks for Deep Visual Representation Learning](https://mlsanity.com/paper/2103.03510/) (2021)
- [BACS: Background Aware Continual Semantic Segmentation](https://mlsanity.com/paper/2404.13148/) (2024)
- [Non-parametric Memory for Spatio-Temporal Segmentation of Construction Zones for Self-Driving](https://mlsanity.com/paper/2101.06865/) (2021)
- [Atlas-aware ConvNetfor Accurate yet Robust Anatomical Segmentation](https://mlsanity.com/paper/2102.01256/) (2021)
- [Empirical Study of Multi-Task Hourglass Model for Semantic Segmentation Task](https://mlsanity.com/paper/2105.13531/) (2021)
- [Deep Neural Networks for Automatic Grain-matrix Segmentation in Plane and Cross-polarized Sandstone Photomicrographs](https://mlsanity.com/paper/2111.07102/) (2021)
- [Instance Segmentation with Cross-Modal Consistency](https://mlsanity.com/paper/2210.08113/) (2022)
- [Deep Dependency Networks for Multi-Label Classification](https://mlsanity.com/paper/2302.00633/) (2023)
- [HR-RCNN: Hierarchical Relational Reasoning for Object Detection](https://mlsanity.com/paper/2110.13892/) (2021)
- [Global and Local Features through Gaussian Mixture Models on Image Semantic Segmentation](https://mlsanity.com/paper/2207.09162/) (2022)
- [Dynamic Graph Message Passing Networks for Visual Recognition](https://mlsanity.com/paper/2209.09760/) (2022)
- [Deep Dependency Networks and Advanced Inference Schemes for Multi-Label Classification](https://mlsanity.com/paper/2404.11667/) (2024)
- [A False Discovery Rate Control Method Using a Fully Connected Hidden Markov Random Field for Neuroimaging Data](https://mlsanity.com/paper/2505.20688/) (2025)
- [UnfoldCRF: Structured Mask Refinement with Image-Conditioned Latent Regions](https://mlsanity.com/paper/2609.33996/) (2026)
- [DeepFlow: Two Convergent Differentiable Approaches for Pairwise Conditional Random Fields](https://mlsanity.com/paper/2610.16546/) (2026)

## Abstract (arXiv)

We address the problem of semantic segmentation using deep learning. Most segmentation systems include a Conditional Random Field (CRF) to produce a structured output that is consistent with the image's visual features. Recent deep learning approaches have incorporated CRFs into Convolutional Neural Networks (CNNs), with some even training the CRF end-to-end with the rest of the network. However, these approaches have not employed higher order potentials, which have previously been shown to significantly improve segmentation performance. In this paper, we demonstrate that two types of higher order potential, based on object detections and superpixels, can be included in a CRF embedded within a deep network. We design these higher order potentials to allow inference with the differentiable mean field algorithm. As a result, all the parameters of our richer CRF model can be learned end-to-end with our pixelwise CNN classifier. We achieve state-of-the-art segmentation performance on the PASCAL VOC benchmark with these trainable higher order potentials.