# Factorized Bilinear Models for Image Recognition

arXiv: https://arxiv.org/abs/1611.05709 · HTML: https://mlsanity.com/paper/1611.05709/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2016.

## Built on by (most cited first, 13 of 13)

- [Fine-Grained Image Analysis with Deep Learning: A Survey](https://mlsanity.com/paper/2111.06119/) (2021)
- [Bilinear MLPs enable weight-based mechanistic interpretability](https://mlsanity.com/paper/2410.08417/) (2024)
- [Augmenting Deep Classifiers with Polynomial Neural Networks](https://mlsanity.com/paper/2104.07916/) (2021)
- [Recurrent Bilinear Optimization for Binary Neural Networks](https://mlsanity.com/paper/2209.01542/) (2022)
- [Part-guided Relational Transformers for Fine-grained Visual Recognition](https://mlsanity.com/paper/2212.13685/) (2022)
- [Verifiable Safety Q-Filters via Hamilton-Jacobi Reachability and Multiplicative Q-Networks](https://mlsanity.com/paper/2506.15693/) (2025)
- [Simulating Weighted Automata over Sequences and Trees with Transformers](https://mlsanity.com/paper/2403.09728/) (2024)
- [Parsing Objects at a Finer Granularity: A Survey](https://mlsanity.com/paper/2212.13693/) (2022)
- [Graph Convolutional Network with Generalized Factorized Bilinear Aggregation](https://mlsanity.com/paper/2107.11666/) (2021)
- [Cogradient Descent for Dependable Learning](https://mlsanity.com/paper/2106.10617/) (2021)
- [A Deeper Look into Second-Order Feature Aggregation for LiDAR Place Recognition](https://mlsanity.com/paper/2409.15919/) (2024)
- [Structural Disentanglement in Bilinear MLPs via Architectural Inductive Bias](https://mlsanity.com/paper/2602.05635/) (2026)
- [Connections Between Pairs of Filters Improve the Accuracy of Convolutional Neural Networks](https://mlsanity.com/paper/2606.13736/) (2026)

## Abstract (arXiv)

Although Deep Convolutional Neural Networks (CNNs) have liberated their power in various computer vision tasks, the most important components of CNN, convolutional layers and fully connected layers, are still limited to linear transformations. In this paper, we propose a novel Factorized Bilinear (FB) layer to model the pairwise feature interactions by considering the quadratic terms in the transformations. Compared with existing methods that tried to incorporate complex non-linearity structures into CNNs, the factorized parameterization makes our FB layer only require a linear increase of parameters and affordable computational cost. To further reduce the risk of overfitting of the FB layer, a specific remedy called DropFactor is devised during the training process. We also analyze the connection between FB layer and some existing models, and show FB layer is a generalization to them. Finally, we validate the effectiveness of FB layer on several widely adopted datasets including CIFAR-10, CIFAR-100 and ImageNet, and demonstrate superior results compared with various state-of-the-art deep models.