# ENACT: Entropy-based Clustering of Attention Input for Reducing the Computational Needs of Object Detection Transformers

arXiv: https://arxiv.org/abs/2409.07541 · HTML: https://mlsanity.com/paper/2409.07541/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 18 references that are in the map.
- First posted on arXiv 11 Sep 2024.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [Fast Transformers with Clustered Attention](https://mlsanity.com/paper/2007.04825/) (2020)
- [End-to-End Object Detection with Adaptive Clustering Transformer](https://mlsanity.com/paper/2011.09315/) (2020)
- [Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformer](https://mlsanity.com/paper/2204.08680/) (2022)
- [MS-DETR: Efficient DETR Training with Mixed Supervision](https://mlsanity.com/paper/2401.03989/) (2024)
- [RDSNet: A New Deep Architecture for Reciprocal Object Detection and Instance Segmentation](https://mlsanity.com/paper/1912.05070/) (2019)

## Built on by (most cited first, 1 of 1)

- [Relieving the Over-Aggregating Effect in Graph Transformers](https://mlsanity.com/paper/2510.21267/) (2025)

## Abstract (arXiv)

Transformers demonstrate competitive performance in terms of precision on the problem of vision-based object detection. However, they require considerable computational resources due to the quadratic size of the attention weights. In this work, we propose to cluster the transformer input on the basis of its entropy, due to its similarity between same object pixels. This is expected to reduce GPU usage during training, while maintaining reasonable accuracy. This idea is realized with an implemented module that is called ENtropy-based Attention Clustering for detection Transformers (ENACT), which serves as a plug-in to any multi-head self-attention based transformer network. Experiments on the COCO object detection dataset and three detection transformers demonstrate that the requirements on memory are reduced, while the detection accuracy is degraded only slightly. The code of the ENACT module is available at https://github.com/GSavathrakis/ENACT.