# Oriented Object Detection with Transformer

arXiv: https://arxiv.org/abs/2106.03146 · HTML: https://mlsanity.com/paper/2106.03146/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 6 Jun 2021.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [DOTA: A Large-scale Dataset for Object Detection in Aerial Images](https://mlsanity.com/paper/1711.10398/) (2017)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [Scaling Neural Machine Translation](https://mlsanity.com/paper/1806.00187/) (2018)
- [ReDet: A Rotation-equivariant Detector for Aerial Object Detection](https://mlsanity.com/paper/2103.07733/) (2021)
- [R3Det: Refined Single-Stage Detector with Feature Refinement for Rotating Object](https://mlsanity.com/paper/1908.05612/) (2019)
- [Align Deep Features for Oriented Object Detection](https://mlsanity.com/paper/2008.09397/) (2020)
- [Arbitrary-Oriented Scene Text Detection via Rotation Proposals](https://mlsanity.com/paper/1703.01086/) (2017)
- [Fast Convergence of DETR with Spatially Modulated Co-Attention](https://mlsanity.com/paper/2101.07448/) (2021)
- [Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering](https://mlsanity.com/paper/1812.05252/) (2018)
- [End-to-End Object Detection with Adaptive Clustering Transformer](https://mlsanity.com/paper/2011.09315/) (2020)
- [R2CNN: Rotational Region CNN for Orientation Robust Scene Text Detection](https://mlsanity.com/paper/1706.09579/) (2017)
- [Dual-stream Network for Visual Recognition](https://mlsanity.com/paper/2105.14734/) (2021)
- [Multi-modality Latent Interaction Network for Visual Question Answering](https://mlsanity.com/paper/1908.04289/) (2019)

## Built on by (most cited first, 20 of 20)

- [The KFIoU Loss for Rotated Object Detection](https://mlsanity.com/paper/2201.12558/) (2022)
- [Transformers in Remote Sensing: A Survey](https://mlsanity.com/paper/2209.01206/) (2022)
- [Poly Kernel Inception Network for Remote Sensing Detection](https://mlsanity.com/paper/2403.06258/) (2024)
- [ARS-DETR: Aspect Ratio-Sensitive Detection Transformer for Aerial Oriented Object Detection](https://mlsanity.com/paper/2303.04989/) (2023)
- [Visualizing and Understanding Patch Interactions in Vision Transformer](https://mlsanity.com/paper/2203.05922/) (2022)
- [Transformers in Small Object Detection: A Benchmark and Survey of State-of-the-Art](https://mlsanity.com/paper/2309.04902/) (2023)
- [M4-SAR: A Multi-Resolution, Multi-Polarization, Multi-Scene, Multi-Source Dataset and Benchmark for optical-SAR Object Detection](https://mlsanity.com/paper/2505.10931/) (2025)
- [Frequency-Dynamic Attention Modulation for Dense Prediction](https://mlsanity.com/paper/2507.12006/) (2025)
- [Hausdorff Distance Matching with Adaptive Query Denoising for Rotated Detection Transformer](https://mlsanity.com/paper/2305.07598/) (2023)
- [Oriented object detection in optical remote sensing images using deep learning: a survey](https://mlsanity.com/paper/2302.10473/) (2023)
- [D2Q-DETR: Decoupling and Dynamic Queries for Oriented Object Detection with Transformers](https://mlsanity.com/paper/2303.00542/) (2023)
- [MutDet: Mutually Optimizing Pre-training for Remote Sensing Object Detection](https://mlsanity.com/paper/2407.09920/) (2024)
- [DPDETR: Decoupled Position Detection Transformer for Infrared-Visible Object Detection](https://mlsanity.com/paper/2408.06123/) (2024)
- [PETDet: Proposal Enhancement for Two-Stage Fine-Grained Object Detection](https://mlsanity.com/paper/2312.10515/) (2023)
- [MKIoU Loss: Towards Accurate Oriented Object Detection in Aerial Images](https://mlsanity.com/paper/2206.15109/) (2022)
- [Bio-Inspired Simple Neural Network for Low-Light Image Restoration: A Minimalist Approach](https://mlsanity.com/paper/2305.01844/) (2023)
- [RotaTR: Detection Transformer for Dense and Rotated Object](https://mlsanity.com/paper/2312.02821/) (2023)
- [Deformable Attention Mechanisms Applied to Object Detection, case of Remote Sensing](https://mlsanity.com/paper/2505.24489/) (2025)
- [PKINet-v2: Towards Powerful and Efficient Poly-Kernel Remote Sensing Object Detection](https://mlsanity.com/paper/2603.16341/) (2026)
- [TCG-AR: Real-Time Multi-View Augmented Reality for Trading Card Game Streaming](https://mlsanity.com/paper/2607.02090/) (2026)

## Abstract (arXiv)

Object detection with Transformers (DETR) has achieved a competitive performance over traditional detectors, such as Faster R-CNN. However, the potential of DETR remains largely unexplored for the more challenging task of arbitrary-oriented object detection problem. We provide the first attempt and implement Oriented Object DEtection with TRansformer (bf O2DETR) based on an end-to-end network. The contributions of rm O2DETR include: 1) we provide a new insight into oriented object detection, by applying Transformer to directly and efficiently localize objects without a tedious process of rotated anchors as in conventional detectors; 2) we design a simple but highly efficient encoder for Transformer by replacing the attention mechanism with depthwise separable convolution, which can significantly reduce the memory and computational cost of using multi-scale features in the original Transformer; 3) our rm O2DETR can be another new benchmark in the field of oriented object detection, which achieves up to 3.85 mAP improvement over Faster R-CNN and RetinaNet. We simply fine-tune the head mounted on rm O2DETR in a cascaded architecture and achieve a competitive performance over SOTA in the DOTA dataset.