# Deep Dependency Networks and Advanced Inference Schemes for Multi-Label Classification

arXiv: https://arxiv.org/abs/2404.11667 · HTML: https://mlsanity.com/paper/2404.11667/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 92 references that are in the map.
- First posted on arXiv 17 Apr 2024.

## Builds on (most cited first, 23 of 23)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [Conditional Random Fields as Recurrent Neural Networks](https://mlsanity.com/paper/1502.03240/) (2015)
- [Deep Kalman Filters](https://mlsanity.com/paper/1511.05121/) (2015)
- [Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation](https://mlsanity.com/paper/1406.2984/) (2014)
- [Query2Label: A Simple Transformer Way to Multi-Label Classification](https://mlsanity.com/paper/2107.10834/) (2021)
- [Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition](https://mlsanity.com/paper/1908.07325/) (2019)
- [Composing graphical models with neural networks for structured representations and fast inference](https://mlsanity.com/paper/1603.06277/) (2016)
- [Deep Convolutional Neural Fields for Depth Estimation from a Single Image](https://mlsanity.com/paper/1411.6387/) (2014)
- [Efficient piecewise training of deep structured models for semantic segmentation](https://mlsanity.com/paper/1504.01013/) (2015)
- [Thin-Slicing Network: A Deep Structured Model for Pose Estimation in Videos](https://mlsanity.com/paper/1703.10898/) (2017)
- [Higher Order Conditional Random Fields in Deep Neural Networks](https://mlsanity.com/paper/1511.08119/) (2015)
- [Articulated Pose Estimation by a Graphical Model with Image Dependent Pairwise Relations](https://mlsanity.com/paper/1407.3399/) (2014)
- [Learning Deep Structured Models](https://mlsanity.com/paper/1407.2538/) (2014)
- [Join-Graph Propagation Algorithms](https://mlsanity.com/paper/1401.3489/) (2014)
- [End-to-End Training of Hybrid CNN-CRF Models for Stereo](https://mlsanity.com/paper/1611.10229/) (2016)
- [Adaptive Graphical Model Network for 2D Handpose Estimation](https://mlsanity.com/paper/1909.08205/) (2019)
- [Multi-layered Semantic Representation Network for Multi-label Image Classification](https://mlsanity.com/paper/2106.11596/) (2021)
- [Joint Training of Generic CNN-CRF Models with Stochastic Optimization](https://mlsanity.com/paper/1511.05067/) (2015)

## Abstract (arXiv)

We present a unified framework called deep dependency networks (DDNs) that combines dependency networks and deep learning architectures for multi-label classification, with a particular emphasis on image and video data. The primary advantage of dependency networks is their ease of training, in contrast to other probabilistic graphical models like Markov networks. In particular, when combined with deep learning architectures, they provide an intuitive, easy-to-use loss function for multi-label classification. A drawback of DDNs compared to Markov networks is their lack of advanced inference schemes, necessitating the use of Gibbs sampling. To address this challenge, we propose novel inference schemes based on local search and integer linear programming for computing the most likely assignment to the labels given observations. We evaluate our novel methods on three video datasets (Charades, TACoS, Wetlab) and three image datasets (MS-COCO, PASCAL VOC, NUS-WIDE), comparing their performance with (a) basic neural architectures and (b) neural architectures combined with Markov networks equipped with advanced inference and learning techniques. Our results demonstrate the superiority of our new DDN methods over the two competing approaches.