# Tensor Factorization for Leveraging Cross-Modal Knowledge in Data-Constrained Infrared Object Detection

arXiv: https://arxiv.org/abs/2309.16592 · HTML: https://mlsanity.com/paper/2309.16592/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 60 references that are in the map.
- First posted on arXiv 28 Sep 2023.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [Scene Graph Generation from Objects, Phrases and Region Captions](https://mlsanity.com/paper/1707.09700/) (2017)
- [Progressive Domain Adaptation for Object Detection](https://mlsanity.com/paper/1910.11319/) (2019)
- [DeepID3: Face Recognition with Very Deep Neural Networks](https://mlsanity.com/paper/1502.00873/) (2015)
- [Uncertainty-Aware Unsupervised Domain Adaptation in Object Detection](https://mlsanity.com/paper/2103.00236/) (2021)
- [Learning Deep Embeddings with Histogram Loss](https://mlsanity.com/paper/1611.00822/) (2016)
- [Domain Adaptation for Object Detection via Style Consistency](https://mlsanity.com/paper/1911.10033/) (2019)
- [Borrow from Anywhere: Pseudo Multi-modal Object Detection in Thermal Imagery](https://mlsanity.com/paper/1905.08789/) (2019)
- [SSDA3D: Semi-supervised Domain Adaptation for 3D Object Detection from Point Cloud](https://mlsanity.com/paper/2212.02845/) (2022)
- [Exploring Low Rank Training of Deep Neural Networks](https://mlsanity.com/paper/2209.13569/) (2022)
- [Multi-Granularity Alignment Domain Adaptation for Object Detection](https://mlsanity.com/paper/2203.16897/) (2022)
- [FCN-rLSTM: Deep Spatio-Temporal Neural Networks for Vehicle Counting in City Cameras](https://mlsanity.com/paper/1707.09476/) (2017)
- [Coreset-Based Neural Network Compression](https://mlsanity.com/paper/1807.09810/) (2018)
- [Multi-Source Domain Adaptation for Object Detection](https://mlsanity.com/paper/2106.15793/) (2021)
- [Unsupervised Domain Adaptation for Object Detection via Cross-Domain Semi-Supervised Learning](https://mlsanity.com/paper/1911.07158/) (2019)
- [Cross-Modal Knowledge Transfer Without Task-Relevant Source Data](https://mlsanity.com/paper/2209.04027/) (2022)
- [Pedestrian Detection in Thermal Images using Saliency Maps](https://mlsanity.com/paper/1904.06859/) (2019)

## Built on by (most cited first, 2 of 2)

- [DKDL-Net: A Lightweight Bearing Fault Detection Model via Decoupled Knowledge Distillation and Low-Rank Adaptation Fine-tuning](https://mlsanity.com/paper/2406.06653/) (2024)
- [Disentangle Object and Non-object Infrared Features via Language Guidance](https://mlsanity.com/paper/2601.09228/) (2026)

## Abstract (arXiv)

The primary bottleneck towards obtaining good recognition performance in IR images is the lack of sufficient labeled training data, owing to the cost of acquiring such data. Realizing that object detection methods for the RGB modality are quite robust (at least for some commonplace classes, like person, car, etc.), thanks to the giant training sets that exist, in this work we seek to leverage cues from the RGB modality to scale object detectors to the IR modality, while preserving model performance in the RGB modality. At the core of our method, is a novel tensor decomposition method called TensorFact which splits the convolution kernels of a layer of a Convolutional Neural Network (CNN) into low-rank factor matrices, with fewer parameters than the original CNN. We first pretrain these factor matrices on the RGB modality, for which plenty of training data are assumed to exist and then augment only a few trainable parameters for training on the IR modality to avoid over-fitting, while encouraging them to capture complementary cues from those trained only on the RGB modality. We validate our approach empirically by first assessing how well our TensorFact decomposed network performs at the task of detecting objects in RGB images vis-a-vis the original network and then look at how well it adapts to IR images of the FLIR ADAS v1 dataset. For the latter, we train models under scenarios that pose challenges stemming from data paucity. From the experiments, we observe that: (i) TensorFact shows performance gains on RGB images; (ii) further, this pre-trained model, when fine-tuned, outperforms a standard state-of-the-art object detector on the FLIR ADAS v1 dataset by about 4% in terms of mAP 50 score.