# A novel multimodal fusion network based on a joint coding model for lane line segmentation

arXiv: https://arxiv.org/abs/2103.11114 · HTML: https://mlsanity.com/paper/2103.11114/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 20 Mar 2021.

## Builds on (most cited first, 16 of 16)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Joint 3D Proposal Generation and Object Detection from View Aggregation](https://mlsanity.com/paper/1712.02294/) (2017)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [Multi-Task Multi-Sensor Fusion for 3D Object Detection](https://mlsanity.com/paper/2012.12397/) (2020)
- [Spatial As Deep: Spatial CNN for Traffic Scene Understanding](https://mlsanity.com/paper/1712.06080/) (2017)
- [Learning Lightweight Lane Detection CNNs by Self Attention Distillation](https://mlsanity.com/paper/1908.00821/) (2019)
- [Towards End-to-End Lane Detection: an Instance Segmentation Approach](https://mlsanity.com/paper/1802.05591/) (2018)
- [Fusion of Multispectral Data Through Illumination-aware Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1802.09972/) (2018)
- [LIDAR-Camera Fusion for Road Detection Using Fully Convolutional Neural Networks](https://mlsanity.com/paper/1809.07941/) (2018)
- [Robust Lane Detection from Continuous Driving Scenes Using Deep Neural Networks](https://mlsanity.com/paper/1903.02193/) (2019)
- [EL-GAN: Embedding Loss Driven Generative Adversarial Networks for Lane Detection](https://mlsanity.com/paper/1806.05525/) (2018)
- [Deep Multi-Sensor Lane Detection](https://mlsanity.com/paper/1905.01555/) (2019)

## Built on by (most cited first, 6 of 6)

- [A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspective](https://mlsanity.com/paper/2405.05173/) (2024)
- [Mix-Teaching: A Simple, Unified and Effective Semi-Supervised Learning Framework for Monocular 3D Object Detection](https://mlsanity.com/paper/2207.04448/) (2022)
- [Informative Data Selection with Uncertainty for Multi-modal Object Detection](https://mlsanity.com/paper/2304.11697/) (2023)
- [Eloss in the way: A Sensitive Input Quality Metrics for Intelligent Driving](https://mlsanity.com/paper/2302.00986/) (2023)
- [Entropy Loss: An Interpretability Amplifier of 3D Object Detection Network for Intelligent Driving](https://mlsanity.com/paper/2409.00839/) (2024)
- [Stabilizing Information Flow Entropy: Regularization for Safe and Interpretable Autonomous Driving Perception](https://mlsanity.com/paper/2509.16277/) (2025)

## Abstract (arXiv)

There has recently been growing interest in utilizing multimodal sensors to achieve robust lane line segmentation. In this paper, we introduce a novel multimodal fusion architecture from an information theory perspective, and demonstrate its practical utility using Light Detection and Ranging (LiDAR) camera fusion networks. In particular, we develop, for the first time, a multimodal fusion network as a joint coding model, where each single node, layer, and pipeline is represented as a channel. The forward propagation is thus equal to the information transmission in the channels. Then, we can qualitatively and quantitatively analyze the effect of different fusion approaches. We argue the optimal fusion architecture is related to the essential capacity and its allocation based on the source and channel. To test this multimodal fusion hypothesis, we progressively determine a series of multimodal models based on the proposed fusion methods and evaluate them on the KITTI and the A2D2 datasets. Our optimal fusion network achieves 85%+ lane line accuracy and 98.7%+ overall. The performance gap among the models will inform continuing future research into development of optimal fusion algorithms for the deep multimodal learning community.