# Regularized Frank-Wolfe for Dense CRFs: Generalizing Mean Field and Beyond

arXiv: https://arxiv.org/abs/2110.14759 · HTML: https://mlsanity.com/paper/2110.14759/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 92 references that are in the map.
- First posted on arXiv 27 Oct 2021.

## Builds on (most cited first, 30 of 38)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1503.01640/) (2015)
- [Differentiation of Blackbox Combinatorial Solvers](https://mlsanity.com/paper/1912.02175/) (2019)
- [Hierarchical Multi-Scale Attention for Semantic Segmentation](https://mlsanity.com/paper/2005.10821/) (2020)
- [Attention to Scale: Scale-aware Semantic Image Segmentation](https://mlsanity.com/paper/1511.03339/) (2015)
- [Simultaneous Detection and Segmentation](https://mlsanity.com/paper/1407.1808/) (2014)
- [On the Global Linear Convergence of Frank-Wolfe Optimization Variants](https://mlsanity.com/paper/1511.05932/) (2015)
- [Efficient piecewise training of deep structured models for semantic segmentation](https://mlsanity.com/paper/1504.01013/) (2015)
- [Not All Pixels Are Equal: Difficulty-aware Semantic Segmentation via Deep Layer Cascade](https://mlsanity.com/paper/1704.01344/) (2017)
- [Semantic Image Segmentation via Deep Parsing Network](https://mlsanity.com/paper/1509.02634/) (2015)
- [A Regularized Framework for Sparse and Structured Neural Attention](https://mlsanity.com/paper/1705.07704/) (2017)
- [The Complexity of Large-scale Convex Programming under a Linear Optimization Oracle](https://mlsanity.com/paper/1309.5550/) (2013)
- [Fast, Exact and Multi-Scale Inference for Semantic Image Segmentation with Deep Gaussian CRFs](https://mlsanity.com/paper/1603.08358/) (2016)
- [Higher Order Conditional Random Fields in Deep Neural Networks](https://mlsanity.com/paper/1511.08119/) (2015)
- [Fully Connected Deep Structured Networks](https://mlsanity.com/paper/1503.02351/) (2015)
- [The Limited Multi-Label Projection Layer](https://mlsanity.com/paper/1906.08707/) (2019)
- [New Analysis and Results for the Frank-Wolfe Method](https://mlsanity.com/paper/1307.0873/) (2013)
- [On learning to localize objects with minimal supervision](https://mlsanity.com/paper/1403.1024/) (2014)
- [Conditional Gradient Algorithms for Norm-Regularized Smooth Convex Optimization](https://mlsanity.com/paper/1302.2325/) (2013)
- [A Unified Optimization View on Generalized Matching Pursuit and Frank-Wolfe](https://mlsanity.com/paper/1702.06457/) (2017)
- [Alternating Direction Graph Matching](https://mlsanity.com/paper/1611.07583/) (2016)
- [Generalized Conditional Gradient for Sparse Estimation](https://mlsanity.com/paper/1410.4828/) (2014)

## Built on by (most cited first, 10 of 10)

- [Loss Functions in the Era of Semantic Segmentation: A Survey and Outlook](https://mlsanity.com/paper/2312.05391/) (2023)
- [INSID3: Training-Free In-Context Segmentation with DINOv3](https://mlsanity.com/paper/2603.28480/) (2026)
- [Modeling Label Correlations for Ultra-Fine Entity Typing with Neural Pairwise Conditional Random Field](https://mlsanity.com/paper/2212.01581/) (2022)
- [Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation](https://mlsanity.com/paper/2311.15211/) (2023)
- [Modeling Label Correlations for Second-Order Semantic Dependency Parsing with Mean-Field Inference](https://mlsanity.com/paper/2204.03619/) (2022)
- [Modeling Instance Interactions for Joint Information Extraction with Neural High-Order Conditional Random Field](https://mlsanity.com/paper/2212.08929/) (2022)
- [CRISP: A Framework for Cryo-EM Image Segmentation and Processing with Conditional Random Field](https://mlsanity.com/paper/2502.08287/) (2025)
- [Scene-Centric Unsupervised Video Panoptic Segmentation](https://mlsanity.com/paper/2606.04925/) (2026)
- [Monotone deep Boltzmann machines](https://mlsanity.com/paper/2307.04990/) (2023)
- [UnfoldCRF: Structured Mask Refinement with Image-Conditioned Latent Regions](https://mlsanity.com/paper/2609.33996/) (2026)

## Abstract (arXiv)

We introduce regularized Frank-Wolfe, a general and effective algorithm for inference and learning of dense conditional random fields (CRFs). The algorithm optimizes a nonconvex continuous relaxation of the CRF inference problem using vanilla Frank-Wolfe with approximate updates, which are equivalent to minimizing a regularized energy function. Our proposed method is a generalization of existing algorithms such as mean field or concave-convex procedure. This perspective not only offers a unified analysis of these algorithms, but also allows an easy way of exploring different variants that potentially yield better performance. We illustrate this in our empirical results on standard semantic segmentation datasets, where several instantiations of our regularized Frank-Wolfe outperform mean field inference, both as a standalone component and as an end-to-end trainable layer in a neural network. We also show that dense CRFs, coupled with our new algorithms, produce significant improvements over strong CNN baselines.