# HybridNets: End-to-End Perception Network

arXiv: https://arxiv.org/abs/2203.09035 · HTML: https://mlsanity.com/paper/2203.09035/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 43 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 22 of 22)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Tversky loss function for image segmentation using 3D fully convolutional deep networks](https://mlsanity.com/paper/1706.05721/) (2017)
- [Spatial As Deep: Spatial CNN for Traffic Scene Understanding](https://mlsanity.com/paper/1712.06080/) (2017)
- [Learning Lightweight Lane Detection CNNs by Self Attention Distillation](https://mlsanity.com/paper/1908.00821/) (2019)
- [MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving](https://mlsanity.com/paper/1612.07695/) (2016)
- [YOLOP: You Only Look Once for Panoptic Driving Perception](https://mlsanity.com/paper/2108.11250/) (2021)
- [LaneNet: Real-Time Lane Detection Networks for Autonomous Driving](https://mlsanity.com/paper/1807.01726/) (2018)
- [BlitzNet: A Real-Time Deep Network for Scene Understanding](https://mlsanity.com/paper/1708.02813/) (2017)
- [LSNet: Extremely Light-Weight Siamese Network For Change Detection in Remote Sensing Image](https://mlsanity.com/paper/2201.09156/) (2022)

## Built on by (most cited first, 30 of 30)

- [Understanding World or Predicting Future? A Comprehensive Survey of World Models](https://mlsanity.com/paper/2411.14499/) (2024)
- [WaterScenes: A Multi-Task 4D Radar-Camera Fusion Dataset and Benchmarks for Autonomous Driving on Water Surfaces](https://mlsanity.com/paper/2307.06505/) (2023)
- [YOLOPv2: Better, Faster, Stronger for Panoptic Driving Perception](https://mlsanity.com/paper/2208.11434/) (2022)
- [Achelous: A Fast Unified Water-surface Panoptic Perception Framework based on Fusion of Monocular Camera and 4D mmWave Radar](https://mlsanity.com/paper/2307.07102/) (2023)
- [You Only Look at Once for Real-time and Generic Multi-Task](https://mlsanity.com/paper/2310.01641/) (2023)
- [TwinLiteNet: An Efficient and Lightweight Model for Driveable Area and Lane Segmentation in Self-Driving Cars](https://mlsanity.com/paper/2307.10705/) (2023)
- [ASY-VRNet: Waterway Panoptic Driving Perception Model based on Asymmetric Fair Fusion of Vision and 4D mmWave Radar](https://mlsanity.com/paper/2308.10287/) (2023)
- [YOLOv1 to YOLOv10: The fastest and most accurate real-time object detection systems](https://mlsanity.com/paper/2408.09332/) (2024)
- [TwinLiteNet+: An Enhanced Multi-Task Segmentation Model for Autonomous Driving](https://mlsanity.com/paper/2403.16958/) (2024)
- [Achelous++: Power-Oriented Water-Surface Panoptic Perception Framework on Edge Devices based on Vision-Radar Fusion and Pruning of Heterogeneous Modalities](https://mlsanity.com/paper/2312.08851/) (2023)
- [Panoptic Perception for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.15388/) (2024)
- [A Survey on Large Language Model-empowered Autonomous Driving](https://mlsanity.com/paper/2409.14165/) (2024)
- [Monocular Lane Detection Based on Deep Learning: A Survey](https://mlsanity.com/paper/2411.16316/) (2024)
- [Prompt-Driven Building Footprint Extraction in Aerial Images with Offset-Building Model](https://mlsanity.com/paper/2310.16717/) (2023)
- [Is Your Autonomous Vehicle Safe? Understanding the Threat of Electromagnetic Signal Injection Attacks on Traffic Scene Perception](https://mlsanity.com/paper/2501.05239/) (2025)
- [RMT-PPAD: Real-time Multi-task Learning for Panoptic Perception in Autonomous Driving](https://mlsanity.com/paper/2508.06529/) (2025)
- [TriLiteNet: Lightweight Model for Multi-Task Visual Perception](https://mlsanity.com/paper/2509.04092/) (2025)
- [UV-M3TL: A Unified and Versatile Multimodal Multi-Task Learning Framework for Assistive Driving Perception](https://mlsanity.com/paper/2602.01594/) (2026)
- [DeepSTEP -- Deep Learning-Based Spatio-Temporal End-To-End Perception for Autonomous Vehicles](https://mlsanity.com/paper/2305.06820/) (2023)
- [Compressing Multi-Task Model for Autonomous Driving via Pruning and Knowledge Distillation](https://mlsanity.com/paper/2511.05557/) (2025)
- [CERBERUS: Simple and Effective All-In-One Automotive Perception Model with Multi Task Learning](https://mlsanity.com/paper/2210.00756/) (2022)
- [ASAP: Accurate semantic segmentation for real time performance](https://mlsanity.com/paper/2210.01323/) (2022)
- [Detection-segmentation convolutional neural network for autonomous vehicle perception](https://mlsanity.com/paper/2306.17485/) (2023)
- [Q-YOLOP: Quantization-aware You Only Look Once for Panoptic Driving Perception](https://mlsanity.com/paper/2307.04537/) (2023)
- [Implementation of a perception system for autonomous vehicles using a detection-segmentation network in SoC FPGA](https://mlsanity.com/paper/2307.08682/) (2023)
- [RainSD: Rain Style Diversification Module for Image Synthesis Enhancement using Feature-Level Style Distribution](https://mlsanity.com/paper/2401.00460/) (2023)
- [A Survey on Deep Multi-Task Learning in Connected Autonomous Vehicles](https://mlsanity.com/paper/2508.00917/) (2025)
- [AurigaNet: A Real-Time Multi-Task Network for Enhanced Urban Driving Perception](https://mlsanity.com/paper/2602.10660/) (2026)
- [DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios](https://mlsanity.com/paper/2605.16892/) (2026)
- [An Exploratory Study of Frequency-Aware Task Weighting for YOLOv8-Based Unified Driving Perception](https://mlsanity.com/paper/2609.05516/) (2026)

## Abstract (arXiv)

End-to-end Network has become increasingly important in multi-tasking. One prominent example of this is the growing significance of a driving perception system in autonomous driving. This paper systematically studies an end-to-end perception network for multi-tasking and proposes several key optimizations to improve accuracy. First, the paper proposes efficient segmentation head and box/class prediction networks based on weighted bidirectional feature network. Second, the paper proposes automatically customized anchor for each level in the weighted bidirectional feature network. Third, the paper proposes an efficient training loss function and training strategy to balance and optimize network. Based on these optimizations, we have developed an end-to-end perception network to perform multi-tasking, including traffic object detection, drivable area segmentation and lane detection simultaneously, called HybridNets, which achieves better accuracy than prior art. In particular, HybridNets achieves 77.3 mean Average Precision on Berkeley DeepDrive Dataset, outperforms lane detection with 31.6 mean Intersection Over Union with 12.83 million parameters and 15.6 billion floating-point operations. In addition, it can perform visual perception tasks in real-time and thus is a practical and accurate solution to the multi-tasking problem. Code is available at https://github.com/datvuthanh/HybridNets.