# BoxeR: Box-Attention for 2D and 3D Transformers

arXiv: https://arxiv.org/abs/2111.13087 · HTML: https://mlsanity.com/paper/2111.13087/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 51 references that are in the map.
- First posted on arXiv 25 Nov 2021.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection](https://mlsanity.com/paper/1912.02424/) (2019)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases](https://mlsanity.com/paper/2103.10697/) (2021)
- [Deep Learning for Generic Object Detection: A Survey](https://mlsanity.com/paper/1809.02165/) (2018)
- [Bottleneck Transformers for Visual Recognition](https://mlsanity.com/paper/2101.11605/) (2021)
- [An End-to-End Transformer Model for 3D Object Detection](https://mlsanity.com/paper/2109.08141/) (2021)

## Built on by (most cited first, 23 of 23)

- [DiffusionDet: Diffusion Model for Object Detection](https://mlsanity.com/paper/2211.09788/) (2022)
- [CenterFormer: Center-based Transformer for 3D Object Detection](https://mlsanity.com/paper/2209.05588/) (2022)
- [FocalFormer3D : Focusing on Hard Instance for 3D Object Detection](https://mlsanity.com/paper/2308.04556/) (2023)
- [Detection Transformer with Stable Matching](https://mlsanity.com/paper/2304.04742/) (2023)
- [Uni3DETR: Unified 3D Detection Transformer](https://mlsanity.com/paper/2310.05699/) (2023)
- [3D Vision with Transformers: A Survey](https://mlsanity.com/paper/2208.04309/) (2022)
- [ConQueR: Query Contrast Voxel-DETR for 3D Object Detection](https://mlsanity.com/paper/2212.07289/) (2022)
- [SEED: A Simple and Effective 3D DETR in Point Clouds](https://mlsanity.com/paper/2407.10749/) (2024)
- [R-MAE: Regions Meet Masked Autoencoders](https://mlsanity.com/paper/2306.05411/) (2023)
- [MUFASA: Multi-View Fusion and Adaptation Network with Spatial Awareness for Radar Object Detection](https://mlsanity.com/paper/2408.00565/) (2024)
- [Li3DeTr: A LiDAR based 3D Detection Transformer](https://mlsanity.com/paper/2210.15365/) (2022)
- [LiDARFormer: A Unified Transformer-based Multi-task Network for LiDAR Perception](https://mlsanity.com/paper/2303.12194/) (2023)
- [One-to-Few Label Assignment for End-to-End Dense Detection](https://mlsanity.com/paper/2303.11567/) (2023)
- [StageInteractor: Query-based Object Detector with Cross-stage Interaction](https://mlsanity.com/paper/2304.04978/) (2023)
- [detrex: Benchmarking Detection Transformers](https://mlsanity.com/paper/2306.07265/) (2023)
- [Spatial-Temporal Graph Enhanced DETR Towards Multi-Frame 3D Object Detection](https://mlsanity.com/paper/2307.00347/) (2023)
- [INSTINCT: Instance-Level Interaction Architecture for Query-Based Collaborative Perception](https://mlsanity.com/paper/2509.23700/) (2025)
- [LaSSM: Efficient Semantic-Spatial Query Decoding via Local Aggregation and State Space Models for 3D Instance Segmentation](https://mlsanity.com/paper/2602.11007/) (2026)
- [ROIFormer: Semantic-Aware Region of Interest Transformer for Efficient Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2212.05729/) (2022)
- [Incorporating Pre-training Data Matters in Unsupervised Domain Adaptation](https://mlsanity.com/paper/2308.03097/) (2023)
- [SPEED: A Scalable RISC-V Vector Processor Enabling Efficient Multi-Precision DNN Inference](https://mlsanity.com/paper/2409.14017/) (2024)
- [Detecting Objects with Context-Likelihood Graphs and Graph Refinement](https://mlsanity.com/paper/2212.12395/) (2022)
- [SimPLR: A Simple and Plain Transformer for Efficient Object Detection and Segmentation](https://mlsanity.com/paper/2310.05920/) (2023)

## Abstract (arXiv)

In this paper, we propose a simple attention mechanism, we call box-attention. It enables spatial interaction between grid features, as sampled from boxes of interest, and improves the learning capability of transformers for several vision tasks. Specifically, we present BoxeR, short for Box Transformer, which attends to a set of boxes by predicting their transformation from a reference window on an input feature map. The BoxeR computes attention weights on these boxes by considering its grid structure. Notably, BoxeR-2D naturally reasons about box information within its attention module, making it suitable for end-to-end instance detection and segmentation tasks. By learning invariance to rotation in the box-attention module, BoxeR-3D is capable of generating discriminative information from a bird's-eye view plane for 3D end-to-end object detection. Our experiments demonstrate that the proposed BoxeR-2D achieves state-of-the-art results on COCO detection and instance segmentation. Besides, BoxeR-3D improves over the end-to-end 3D object detection baseline and already obtains a compelling performance for the vehicle category of Waymo Open, without any class-specific optimization. Code is available at https://github.com/kienduynguyen/BoxeR.