mlsanityData to 7 Oct 2026

Video Task Decathlon: Unifying Image and Video Tasks in Autonomous Driving

arXiv 2309.04422 ↗First posted 8 Sep 2023cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Performing multiple heterogeneous visual tasks in dynamic scenes is a hallmark of human perception capability. Despite remarkable progress in image and video recognition via representation learning, current research still focuses on designing specialized networks for singular, homogeneous, or simple combination of tasks. We instead explore the construction of a unified model for major image and video recognition tasks in autonomous driving with diverse input and output structures. To enable such an investigation, we design a new challenge, Video Task Decathlon (VTD), which includes ten representative image and video tasks spanning classification, segmentation, localization, and association of objects and pixels. On VTD, we develop our unified network, VTDNet, that uses a single structure and a single set of weights for all ten tasks. VTDNet groups similar tasks and employs task interaction stages to exchange information within and between task groups. Given the impracticality of labeling all tasks on all frames, and the performance degradation associated with joint training of many tasks, we design a Curriculum training, Pseudo-labeling, and Fine-tuning (CPF) scheme to successfully train VTDNet on all tasks and mitigate performance loss. Armed with CPF, VTDNet significantly outperforms its single-task counterparts on most tasks with only 20% overall computations. VTD is a promising new direction for exploring the unification of perception tasks in autonomous driving.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 66 of its 83 references that are in the map. First posted on arXiv 8 Sep 2023.

Builds on

66 of 83 references.

References · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2017Mask R-CNNcited by 7,259
  7. 2016Layer Normalizationcited by 5,579
  8. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  10. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  11. 2022A ConvNet for the 2020scited by 3,545
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
38 more
  1. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  2. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  4. 2018Group Normalizationcited by 1,514
  5. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  6. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  7. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  8. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  9. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  10. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  11. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  12. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  13. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  14. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  15. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  16. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  17. 2016Performance Measures and a Data Set for Multi-Target, Multi-Camera Trackingcited by 685
  18. 2018Deformable ConvNets v2: More Deformable, Better Resultscited by 666
  19. 2019Panoptic Feature Pyramid Networkscited by 644
  20. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  21. 2017Learning multiple visual domains with residual adapterscited by 553
  22. 2020HOTA: A Higher Order Metric for Evaluating Multi-Object Trackingcited by 546
  23. 2016MOT16: A Benchmark for Multi-Object Trackingcited by 531
  24. 2018Simple Baselines for Human Pose Estimation and Trackingcited by 502
  25. 2019Hybrid Task Cascade for Instance Segmentationcited by 479
  26. 2020MOT20: A benchmark for multi object tracking in crowded scenescited by 355
  27. 2018PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsingcited by 224
  28. 2020Quasi-Dense Similarity Learning for Multiple Object Trackingcited by 220
  29. 2017UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Losscited by 202
  30. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  31. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 159
  32. 2020MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learningcited by 157
  33. 2019Attentive Single-Tasking of Multiple Taskscited by 148
  34. 2022SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptationcited by 146
  35. 2019Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentationcited by 141
  36. 2020Disentangled Non-Local Neural Networkscited by 132
  37. 2017Occlusion Aware Unsupervised Learning of Optical Flowcited by 101
  38. 2022Towards Grand Unification of Object Trackingcited by 93

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 2; 2026: 1.

8 papers.

  1. 2024A Backpack Full of Skills: Egocentric Video Understanding with Diverse Task Perspectivescited by 9
  2. 2025Hier-EgoPack: Hierarchical Egocentric Video Understanding with Diverse Task Perspectivescited by 5
  3. 2024JDT3D: Addressing the Gaps in LiDAR-Based Tracking-by-Attentioncited by 4
  4. 2024CycleCrash: A Dataset of Bicycle Collision Videos for Collision Prediction and Analysiscited by 3
  5. 2024FusionSAM: Visual Multi-Modal Learning with Segment Anythingcited by 1
  6. 2025Learning with Preserving for Continual Multitask Learningcited by 1
  7. 2024DLM-VMTL:A Double Layer Mapper for heterogeneous data video Multi-task prompt learningcited by 0
  8. 2026Unified Video Dense Prediction from Disjoint Datacited by 0