# Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernels

arXiv: https://arxiv.org/abs/1908.09597 · HTML: https://mlsanity.com/paper/1908.09597/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Aug 2019.

## Built on by (most cited first, 30 of 42)

- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning](https://mlsanity.com/paper/2110.01515/) (2021)
- [FAMO: Fast Adaptive Multitask Optimization](https://mlsanity.com/paper/2306.03792/) (2023)
- [Towards Modular LLMs by Building and Reusing a Library of LoRAs](https://mlsanity.com/paper/2405.11157/) (2024)
- [Recon: Reducing Conflicting Gradients from the Root for Multi-Task Learning](https://mlsanity.com/paper/2302.11289/) (2023)
- [DINO-Foresight: Looking into the Future with DINO](https://mlsanity.com/paper/2412.11673/) (2024)
- [Learning Multiple Dense Prediction Tasks from Partially Annotated Data](https://mlsanity.com/paper/2111.14893/) (2021)
- [Multimodal Instruction Tuning with Conditional Mixture of LoRA](https://mlsanity.com/paper/2402.15896/) (2024)
- [Universal Representations: A Unified Look at Multiple Task and Domain Learning](https://mlsanity.com/paper/2204.02744/) (2022)
- [E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection](https://mlsanity.com/paper/2403.09323/) (2024)
- [Scalarization for Multi-Task and Multi-Domain Learning at Scale](https://mlsanity.com/paper/2310.08910/) (2023)
- [Mitigating Task Interference in Multi-Task Learning via Explicit Task Routing with Non-Learnable Primitives](https://mlsanity.com/paper/2308.02066/) (2023)
- [AutoMTL: A Programming Framework for Automating Efficient Multi-Task Learning](https://mlsanity.com/paper/2110.13076/) (2021)
- [Variational Multi-Task Learning with Gumbel-Softmax Priors](https://mlsanity.com/paper/2111.05323/) (2021)
- [Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learners](https://mlsanity.com/paper/2212.08066/) (2022)
- [Prompt Guided Transformer for Multi-Task Dense Prediction](https://mlsanity.com/paper/2307.15362/) (2023)
- [GRIDS: Grouped Multiple-Degradation Restoration with Image Degradation Similarity](https://mlsanity.com/paper/2407.12273/) (2024)
- [Multi-task Learning with 3D-Aware Regularization](https://mlsanity.com/paper/2310.00986/) (2023)
- [Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning](https://mlsanity.com/paper/2410.14633/) (2024)
- [Multi-Task Learning for Visual Scene Understanding](https://mlsanity.com/paper/2203.14896/) (2022)
- [When Multi-Task Learning Meets Partial Supervision: A Computer Vision Review](https://mlsanity.com/paper/2307.14382/) (2023)
- [Joint-Task Regularization for Partially Labeled Multi-Task Learning](https://mlsanity.com/paper/2404.01976/) (2024)
- [Automated Multi-Task Learning for Joint Disease Prediction on Electronic Health Records](https://mlsanity.com/paper/2403.04086/) (2024)
- [Instance-Level Task Parameters: A Robust Multi-task Weighting Framework](https://mlsanity.com/paper/2106.06129/) (2021)
- [Improved optimization strategies for deep Multi-Task Networks](https://mlsanity.com/paper/2109.11678/) (2021)
- [Resolving Token-Space Gradient Conflicts: Token Space Manipulation for Transformer-Based Multi-Task Learning](https://mlsanity.com/paper/2507.07485/) (2025)
- [Feature Pyramid Network for Multi-task Affective Analysis](https://mlsanity.com/paper/2107.03670/) (2021)
- [Revisit the Imbalance Optimization in Multi-task Learning: An Experimental Analysis](https://mlsanity.com/paper/2509.23915/) (2025)
- [Multi-Task Structural Learning using Local Task Similarity induced Neuron Creation and Removal](https://mlsanity.com/paper/2305.00441/) (2023)
- [Nested Hash Layer: A Plug-and-play Module for Multiple-length Hash Code Learning](https://mlsanity.com/paper/2412.08922/) (2024)

## Abstract (arXiv)

The performance of multi-task learning in Convolutional Neural Networks (CNNs) hinges on the design of feature sharing between tasks within the architecture. The number of possible sharing patterns are combinatorial in the depth of the network and the number of tasks, and thus hand-crafting an architecture, purely based on the human intuitions of task relationships can be time-consuming and suboptimal. In this paper, we present a probabilistic approach to learning task-specific and shared representations in CNNs for multi-task learning. Specifically, we propose "stochastic filter groups'' (SFG), a mechanism to assign convolution kernels in each layer to "specialist'' or "generalist'' groups, which are specific to or shared across different tasks, respectively. The SFG modules determine the connectivity between layers and the structures of task-specific and shared representations in the network. We employ variational inference to learn the posterior distribution over the possible grouping of kernels and network parameters. Experiments demonstrate that the proposed method generalises across multiple tasks and shows improved performance over baseline methods.