# An Exploratory Study of Frequency-Aware Task Weighting for YOLOv8-Based Unified Driving Perception

arXiv: https://arxiv.org/abs/2609.05516 · HTML: https://mlsanity.com/paper/2609.05516/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 29 references that are in the map.
- First posted on arXiv 31 Aug 2026.

## Builds on (most cited first, 24 of 24)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing](https://mlsanity.com/paper/1805.04409/) (2018)
- [Multi-Task Learning as a Bargaining Game](https://mlsanity.com/paper/2202.01017/) (2022)
- [Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art](https://mlsanity.com/paper/1704.05519/) (2017)
- [MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning](https://mlsanity.com/paper/2001.06902/) (2020)
- [MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving](https://mlsanity.com/paper/1612.07695/) (2016)
- [YOLOP: You Only Look Once for Panoptic Driving Perception](https://mlsanity.com/paper/2108.11250/) (2021)
- [A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms](https://mlsanity.com/paper/2009.06125/) (2020)
- [HybridNets: End-to-End Perception Network](https://mlsanity.com/paper/2203.09035/) (2022)
- [YOLOPv2: Better, Faster, Stronger for Panoptic Driving Perception](https://mlsanity.com/paper/2208.11434/) (2022)
- [Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspective](https://mlsanity.com/paper/2110.06256/) (2021)

## Abstract (arXiv)

Unified perception enables autonomous driving systems to perform object detection, drivable-area segmentation, and lane segmentation within a single network, improving efficiency and reducing deployment complexity. Jointly optimizing multiple perception tasks remains challenging because tasks exhibit different convergence rates, loss scales, and optimization stability. Existing task-weighting methods use loss magnitude, learned uncertainty, short-term loss changes, or gradient statistics; here, we explore the frequency structure of a recent loss-history window as a complementary signal. We implement and examine Frequency-aware Task Weighting (FTW), a dynamic task-balancing rule that estimates a loss-trajectory stability proxy from the low-frequency energy ratio of recent loss histories. FTW assigns larger weights to tasks whose mean-centered loss trajectories contain a larger proportion of low-frequency power. We document FTW and two baselines under full-network static training and progressive freezing using a unified YOLOv8-based perception framework with three task-specific heads. Experiments on Mapillary Vistas compare FTW with fixed and uncertainty-based weighting under both configurations. Final holdout metrics are reported for the checkpoint with the lowest per-epoch validation loss in each run. Across six single-run configurations, static FTW has the largest derived overall score and lane mIoU, progressive FTW has the largest detection mAP, and static uncertainty weighting has the largest drivable-area mIoU. Without repeated-seed estimates, single-task baselines, or FTW ablations, these rankings are descriptive. The evidence supports the feasibility of loss-frequency-based weighting in this pipeline, but does not establish improvement over the baselines or generalization beyond the reported runs.