# Miti-DETR: Object Detection based on Transformers with Mitigatory Self-Attention Convergence

arXiv: https://arxiv.org/abs/2112.13310 · HTML: https://mlsanity.com/paper/2112.13310/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 34 references that are in the map.
- First posted on arXiv 26 Dec 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [A Decomposable Attention Model for Natural Language Inference](https://mlsanity.com/paper/1606.01933/) (2016)
- [Long Short-Term Memory-Networks for Machine Reading](https://mlsanity.com/paper/1601.06733/) (2016)
- [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://mlsanity.com/paper/1911.08460/) (2019)
- [Structured Attention Networks](https://mlsanity.com/paper/1702.00887/) (2017)
- [MDFN: Multi-Scale Deep Feature Learning Network for Object Detection](https://mlsanity.com/paper/1912.04514/) (2019)
- [SGNet: A Super-class Guided Network for Image Classification and Object Detection](https://mlsanity.com/paper/2104.12898/) (2021)
- [Audio-Visual Transformer Based Crowd Counting](https://mlsanity.com/paper/2109.01926/) (2021)
- [Efficient Golf Ball Detection and Tracking Based on Convolutional Neural Networks and Kalman Filter](https://mlsanity.com/paper/2012.09393/) (2020)
- [Six-channel Image Representation for Cross-domain Object Detection](https://mlsanity.com/paper/2101.00561/) (2021)
- [A Fine-Grained Visual Attention Approach for Fingerspelling Recognition in the Wild](https://mlsanity.com/paper/2105.07625/) (2021)
- [Semantic Clustering based Deduction Learning for Image Recognition and Classification](https://mlsanity.com/paper/2112.13165/) (2021)
- [Location-Aware Box Reasoning for Anchor-Based Single-Shot Object Detection](https://mlsanity.com/paper/2007.06233/) (2020)
- [Lipschitz Networks and Distributional Robustness](https://mlsanity.com/paper/1809.01129/) (2018)
- [Parallel Scale-wise Attention Network for Effective Scene Text Recognition](https://mlsanity.com/paper/2104.12076/) (2021)

## Built on by (most cited first, 12 of 12)

- [Accumulated Trivial Attention Matters in Vision Transformers on Small Datasets](https://mlsanity.com/paper/2210.12333/) (2022)
- [Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets](https://mlsanity.com/paper/2407.19394/) (2024)
- [Aggregating Global Features into Local Vision Transformer](https://mlsanity.com/paper/2201.12903/) (2022)
- [Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapse](https://mlsanity.com/paper/2505.16284/) (2025)
- [Dynamic Label Assignment for Object Detection by Combining Predicted IoUs and Anchor IoUs](https://mlsanity.com/paper/2201.09396/) (2022)
- [Aphid Cluster Recognition and Detection in the Wild Using Deep Learning Models](https://mlsanity.com/paper/2308.05881/) (2023)
- [PF3Det: A Prompted Foundation Feature Assisted Visual LiDAR 3D Detector](https://mlsanity.com/paper/2504.03563/) (2025)
- [A New Dataset and Comparative Study for Aphid Cluster Detection](https://mlsanity.com/paper/2307.05929/) (2023)
- [Gender, Smoking History and Age Prediction from Laryngeal Images](https://mlsanity.com/paper/2305.16661/) (2023)
- [Explicitly Increasing Input Information Density for Vision Transformers on Small Datasets](https://mlsanity.com/paper/2210.14319/) (2022)
- [Object Detection for Vehicle Dashcams using Transformers](https://mlsanity.com/paper/2408.15809/) (2024)
- [Beyond Isolated Heads: Multi-Overlapped-Head Self-Attention for Vision Transformers](https://mlsanity.com/paper/2410.14874/) (2024)

## Abstract (arXiv)

Object Detection with Transformers (DETR) and related works reach or even surpass the highly-optimized Faster-RCNN baseline with self-attention network architectures. Inspired by the evidence that pure self-attention possesses a strong inductive bias that leads to the transformer losing the expressive power with respect to network depth, we propose a transformer architecture with a mitigatory self-attention mechanism by applying possible direct mapping connections in the transformer architecture to mitigate the rank collapse so as to counteract feature expression loss and enhance the model performance. We apply this proposal in object detection tasks and develop a model named Miti-DETR. Miti-DETR reserves the inputs of each single attention layer to the outputs of that layer so that the "non-attention" information has participated in any attention propagation. The formed residual self-attention network addresses two critical issues: (1) stop the self-attention networks from degenerating to rank-1 to the maximized degree; and (2) further diversify the path distribution of parameter update so that easier attention learning is expected. Miti-DETR significantly enhances the average detection precision and convergence speed towards existing DETR-based models on the challenging COCO object detection dataset. Moreover, the proposed transformer with the residual self-attention network can be easily generalized or plugged in other related task models without specific customization.