# MSNN-LINet: Cross-Modal Learning via Continuous Linear Integration

arXiv: https://arxiv.org/abs/2606.31135 · HTML: https://mlsanity.com/paper/2606.31135/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 30 Jun 2026.

## Builds on (most cited first, 12 of 12)

- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Depth-aware CNN for RGB-D Segmentation](https://mlsanity.com/paper/1803.06791/) (2018)
- [DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation](https://mlsanity.com/paper/2309.09668/) (2023)
- [Translate-to-Recognize Networks for RGB-D Scene Recognition](https://mlsanity.com/paper/1904.12254/) (2019)
- [RFBNet: Deep Multimodal Networks with Residual Fusion Blocks for RGB-D Semantic Segmentation](https://mlsanity.com/paper/1907.00135/) (2019)
- [MMP: Towards Robust Multi-Modal Learning with Masked Modality Projection](https://mlsanity.com/paper/2410.03010/) (2024)
- [CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasets](https://mlsanity.com/paper/2302.06148/) (2023)
- [Depth CNNs for RGB-D scene recognition: learning from scratch better than transferring from RGB-CNNs](https://mlsanity.com/paper/1801.06797/) (2018)
- [Learning Contrastive Multimodal Fusion with Improved Modality Dropout for Disease Detection and Prediction](https://mlsanity.com/paper/2509.18284/) (2025)
- [Negative to Positive Co-learning with Aggressive Modality Dropout](https://mlsanity.com/paper/2501.00865/) (2025)
- [Robust Multimodal Learning via Entropy-Gated Contrastive Fusion](https://mlsanity.com/paper/2505.15417/) (2025)

## Abstract (arXiv)

We present LINet (Linear Integration Network), a Multi-Stream Neural Network (MSNN) for RGB-D scene classification. Current multi-modal architectures treat feature fusion as a discrete, ad-hoc event: early fusion entangles representations prematurely, late fusion isolates them until the final layer, and hybrid or attention-based methods require architectural guesswork to place intermediate fusion blocks. LINet addresses this structural compromise by maintaining three dedicated parallel streams (RGB, depth, and integration) where a novel Linear Integration Convolution (LIConv2d) operator enables continuous cross-modal learning at every layer. The integration stream receives raw filtered signals from both modality streams and combines them before the nonlinear activation threshold, conceptually inspired by somatic integration preceding the neuronal firing decision. Implementing continuous integration exposes a critical initialization pathology: Kaiming initialization of the bridging weights scrambles gradients before they reach the stream backbones, producing a failure mode that resembles overfitting but is corrupted gradient flow. A 1/N constant initialization mitigates this. We employ progressive modality dropout, a curriculum adapted to continuous fusion in which blanking probability increases from zero, preventing pathway collapse, a form of negative co-learning, by forcing robust independent stream representations. Trained from scratch on SUN RGB-D 19-class scene classification, LINet reaches 45.2% mean class accuracy at ResNet18 scale, outperforming prior from-scratch results, and rises to 49.6% with in-domain RGB-D (ScanNet) pretraining.