# YOLO-MS: Rethinking Multi-Scale Representation Learning for Real-time Object Detection

arXiv: https://arxiv.org/abs/2308.05480 · HTML: https://mlsanity.com/paper/2308.05480/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 68 of its 86 references that are in the map.
- First posted on arXiv 10 Aug 2023.

## Builds on (most cited first, 30 of 68)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)

## Built on by (most cited first, 30 of 33)

- [YOLOv10: Real-Time End-to-End Object Detection](https://mlsanity.com/paper/2405.14458/) (2024)
- [YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information](https://mlsanity.com/paper/2402.13616/) (2024)
- [YOLOv12: Attention-Centric Real-Time Object Detectors](https://mlsanity.com/paper/2502.12524/) (2025)
- [DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation](https://mlsanity.com/paper/2309.09668/) (2023)
- [YOLOv13: Real-Time Object Detection with Hypergraph-Enhanced Adaptive Visual Perception](https://mlsanity.com/paper/2506.17733/) (2025)
- [SARDet-100K: Towards Open-Source Benchmark and ToolKit for Large-Scale SAR Object Detection](https://mlsanity.com/paper/2403.06534/) (2024)
- [LSKNet: A Foundation Lightweight Backbone for Remote Sensing](https://mlsanity.com/paper/2403.11735/) (2024)
- [LW-DETR: A Transformer Replacement to YOLO for Real-Time Detection](https://mlsanity.com/paper/2406.03459/) (2024)
- [RT-DETRv3: Real-time End-to-End Object Detection with Hierarchical Dense Positive Supervision](https://mlsanity.com/paper/2409.08475/) (2024)
- [Hyper-YOLO: When Visual Object Detection Meets Hypergraph Computation](https://mlsanity.com/paper/2408.04804/) (2024)
- [YOLOv8 to YOLO11: A Comprehensive Architecture In-depth Comparative Review](https://mlsanity.com/paper/2501.13400/) (2025)
- [Strip R-CNN: Large Strip Convolution for Remote Sensing Object Detection](https://mlsanity.com/paper/2501.03775/) (2025)
- [ATRNet-STAR: A Large Dataset and Benchmark Towards Remote Sensing Object Recognition in the Wild](https://mlsanity.com/paper/2501.13354/) (2025)
- [Towards RAW Object Detection in Diverse Conditions](https://mlsanity.com/paper/2411.15678/) (2024)
- [Multi-Branch Auxiliary Fusion YOLO with Re-parameterization Heterogeneous Convolutional for accurate object detection](https://mlsanity.com/paper/2407.04381/) (2024)
- [MHAF-YOLO: Multi-Branch Heterogeneous Auxiliary Fusion YOLO for accurate object detection](https://mlsanity.com/paper/2502.04656/) (2025)
- [Self-Adaptive Gamma Context-Aware SSM-based Model for Metal Defect Detection](https://mlsanity.com/paper/2503.01234/) (2025)
- [BeltCrack: the First Sequential-image Industrial Conveyor Belt Crack Detection Dataset and Its Baseline with Triple-domain Feature Learning](https://mlsanity.com/paper/2506.17892/) (2025)
- [Real-Time Oriented Object Detection Transformer in Remote Sensing Images](https://mlsanity.com/paper/2603.15497/) (2026)
- [DASSF: Dynamic-Attention Scale-Sequence Fusion for Aerial Object Detection](https://mlsanity.com/paper/2406.12285/) (2024)
- [OG-HFYOLO :Orientation gradient guidance and heterogeneous feature fusion for deformation table cell instance segmentation](https://mlsanity.com/paper/2504.20682/) (2025)
- [Teaching in adverse scenes: a statistically feedback-driven threshold and mask adjustment teacher-student framework for object detection in UAV images under adverse scenes](https://mlsanity.com/paper/2506.11175/) (2025)
- [MSMVD: Exploiting Multi-scale Image Features via Multi-scale BEV Features for Multi-view Pedestrian Detection](https://mlsanity.com/paper/2508.20447/) (2025)
- [DEF-YOLO: Leveraging YOLO for Concealed Weapon Detection in Thermal Imagin](https://mlsanity.com/paper/2510.13326/) (2025)
- [Exploring Scale Shift in Crowd Localization under the Context of Domain Generalization](https://mlsanity.com/paper/2510.19330/) (2025)
- [RadHARSimulator V2: Video to Doppler Generator](https://mlsanity.com/paper/2511.09022/) (2025)
- [WUTDet: A 100K-Scale Ship Detection Dataset and Benchmarks with Dense Small Objects](https://mlsanity.com/paper/2604.07759/) (2026)
- [DPTrack:Directional Kernel-Guided Prompt Learning for Robust Nighttime Aerial Tracking](https://mlsanity.com/paper/2510.15449/) (2025)
- [Knowledge-Informed Neural Network for Complex-Valued SAR Image Recognition](https://mlsanity.com/paper/2510.20284/) (2025)
- [You Only Look Omni Gradient Backpropagation for Moving Infrared Small Target Detection](https://mlsanity.com/paper/2511.13013/) (2025)

## Abstract (arXiv)

We aim at providing the object detection community with an efficient and performant object detector, termed YOLO-MS. The core design is based on a series of investigations on how multi-branch features of the basic block and convolutions with different kernel sizes affect the detection performance of objects at different scales. The outcome is a new strategy that can significantly enhance multi-scale feature representations of real-time object detectors. To verify the effectiveness of our work, we train our YOLO-MS on the MS COCO dataset from scratch without relying on any other large-scale datasets, like ImageNet or pre-trained weights. Without bells and whistles, our YOLO-MS outperforms the recent state-of-the-art real-time object detectors, including YOLO-v7, RTMDet, and YOLO-v8. Taking the XS version of YOLO-MS as an example, it can achieve an AP score of 42+% on MS COCO, which is about 2% higher than RTMDet with the same model size. Furthermore, our work can also serve as a plug-and-play module for other YOLO models. Typically, our method significantly advances the APs, APl, and AP of YOLOv8-N from 18%+, 52%+, and 37%+ to 20%+, 55%+, and 40%+, respectively, with even fewer parameters and MACs. Code and trained models are publicly available at https://github.com/FishAndWasabi/YOLO-MS. We also provide the Jittor version at https://github.com/NK-JittorCV/nk-yolo.