mlsanityData to 7 Oct 2026

YOLOR-Based Multi-Task Learning

arXiv 2309.16921 ↗First posted 29 Sep 2023cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Multi-task learning (MTL) aims to learn multiple tasks using a single model and jointly improve all of them assuming generalization and shared semantics. Reducing conflicts between tasks during joint learning is difficult and generally requires careful network design and extremely large models. We propose building on You Only Learn One Representation (YOLOR), a network architecture specifically designed for multitasking. YOLOR leverages both explicit and implicit knowledge, from data observations and learned latents, respectively, to improve a shared representation while minimizing the number of training parameters. However, YOLOR and its follow-up, YOLOv7, only trained two tasks at once. In this paper, we jointly train object detection, instance segmentation, semantic segmentation, and image captioning. We analyze tradeoffs and attempt to maximize sharing of semantic information. Through our architecture and training strategies, we find that our method achieves competitive performance on all tasks while maintaining a low parameter count and without any pre-training. We will release code soon.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 35 of its 44 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

35 of 44 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  10. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  11. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  12. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
23 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  2. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  3. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  4. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  5. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  6. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  7. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  8. 2017A Survey on Multi-Task Learningcited by 838
  9. 2018Panoptic Segmentationcited by 796
  10. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  11. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  12. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  13. 2019YOLACT: Real-time Instance Segmentationcited by 441
  14. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 374
  15. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 218
  16. 2022A Unified Sequence Interface for Vision Taskscited by 152
  17. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  18. 2016HyperFace: A Deep Multi-task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognitioncited by 103
  19. 2021You Only Learn One Representation: Unified Network for Multiple Taskscited by 86
  20. 2016Trace Norm Regularised Deep Multi-Task Learningcited by 50
  21. 2022Designing Network Design Strategies Through Gradient Path Analysiscited by 32
  22. 2019On Better Exploring and Exploiting Task Relationships in Multi-Task Learning: Joint Model and Feature Learningcited by 3
  23. 2022Image Captioning In the Transformer Agecited by 3

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 2.

5 papers.

  1. 2024Finetuning YOLOv9 for Vehicle Detection: Deep Learning for Intelligent Transportation Systems in Dhaka, Bangladeshcited by 2
  2. 2024Temporal2Seq: A Unified Framework for Temporal Video Understanding Taskscited by 1
  3. 2024Surveying You Only Look Once (YOLO) Multispectral Object Detection Advancements, Applications And Challengescited by 0
  4. 2025SSVQ: Unleashing the Potential of Vector Quantization with Sign-Splittingcited by 0
  5. 2025Beyond Anonymization: Object Scrubbing for Privacy-Preserving 2D and 3D Vision Taskscited by 0