# Asymmetric 3D Context Fusion for Universal Lesion Detection

arXiv: https://arxiv.org/abs/2109.08684 · HTML: https://mlsanity.com/paper/2109.08684/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 17 Sep 2021.

## Builds on (most cited first, 21 of 21)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis](https://mlsanity.com/paper/2010.14925/) (2020)
- [Deep Lesion Graphs in the Wild: Relationship Learning and Organization of Significant Radiology Image Findings in a Diverse Large-scale Lesion Database](https://mlsanity.com/paper/1711.10535/) (2017)
- [MULAN: Multitask Universal Lesion Analysis Network for Joint Lesion Detection, Tagging, and Segmentation](https://mlsanity.com/paper/1908.04373/) (2019)
- [Improving RetinaNet for CT Lesion Detection with Dense Masks from Weak RECIST Labels](https://mlsanity.com/paper/1906.02283/) (2019)
- [Volumetric Attention for 3D Medical Image Segmentation and Detection](https://mlsanity.com/paper/2004.01997/) (2020)
- [3D Context Enhanced Region-based Convolutional Neural Network for End-to-End Lesion Detection](https://mlsanity.com/paper/1806.09648/) (2018)
- [ULDor: A Universal Lesion Detector for CT Scans with Pseudo Masks and Hard Negative Example Mining](https://mlsanity.com/paper/1901.06359/) (2019)
- [MVP-Net: Multi-view FPN with Position-aware Attention for Deep Universal Lesion Detection](https://mlsanity.com/paper/1909.04247/) (2019)
- [AlignShift: Bridging the Gap of Imaging Thickness in 3D Anisotropic Volumes](https://mlsanity.com/paper/2005.01969/) (2020)
- [Revisiting 3D Context Modeling with Supervised Pre-training for Universal Lesion Detection in CT Slices](https://mlsanity.com/paper/2012.08770/) (2020)
- [Bounding Maps for Universal Lesion Detection](https://mlsanity.com/paper/2007.09383/) (2020)

## Built on by (most cited first, 14 of 14)

- [SATr: Slice Attention with Transformer for Universal Lesion Detection](https://mlsanity.com/paper/2203.07373/) (2022)
- [CancerUniT: Towards a Single Unified Model for Effective Detection, Segmentation, and Diagnosis of Eight Major Cancers Using a Large Collection of CT Scans](https://mlsanity.com/paper/2301.12291/) (2023)
- [Deep Rib Fracture Instance Segmentation and Classification from CT on the RibFrac Challenge](https://mlsanity.com/paper/2402.09372/) (2024)
- [View-Disentangled Transformer for Brain Lesion Detection](https://mlsanity.com/paper/2209.09657/) (2022)
- [Effective Lymph Nodes Detection in CT Scans Using Location Debiased Query Selection and Contrastive Query Representation in Transformer](https://mlsanity.com/paper/2404.03819/) (2024)
- [Tackling the Incomplete Annotation Issue in Universal Lesion Detection Task By Exploratory Training](https://mlsanity.com/paper/2309.13306/) (2023)
- [Class Imbalance Correction for Improved Universal Lesion Detection and Tagging in CT](https://mlsanity.com/paper/2504.05591/) (2025)
- [Mixed-order self-paced curriculum learning for universal lesion detection](https://mlsanity.com/paper/2302.04677/) (2023)
- [Pay Less On Clinical Images: Asymmetric Multi-Modal Fusion Method For Efficient Multi-Label Skin Lesion Classification](https://mlsanity.com/paper/2407.09999/) (2024)
- [From Slices to Sequences: Autoregressive Tracking Transformer for Cohesive and Consistent 3D Lymph Node Detection in CT Scans](https://mlsanity.com/paper/2503.07933/) (2025)
- [3D Universal Lesion Detection and Tagging in CT with Self-Training](https://mlsanity.com/paper/2504.05201/) (2025)
- [Correcting Class Imbalances with Self-Training for Improved Universal Lesion Detection and Tagging](https://mlsanity.com/paper/2504.05207/) (2025)
- [MoMBS: Mixed-order minibatch sampling enhances model training from diverse-quality images](https://mlsanity.com/paper/2505.18741/) (2025)
- [PlaneCycle: Training-Free 2D-to-3D Lifting of Foundation Models Without Adapters](https://mlsanity.com/paper/2603.04165/) (2026)

## Abstract (arXiv)

Modeling 3D context is essential for high-performance 3D medical image analysis. Although 2D networks benefit from large-scale 2D supervised pretraining, it is weak in capturing 3D context. 3D networks are strong in 3D context yet lack supervised pretraining. As an emerging technique, 3D context fusion operator, which enables conversion from 2D pretrained networks, leverages the advantages of both and has achieved great success. Existing 3D context fusion operators are designed to be spatially symmetric, i.e., performing identical operations on each 2D slice like convolutions. However, these operators are not truly equivariant to translation, especially when only a few 3D slices are used as inputs. In this paper, we propose a novel asymmetric 3D context fusion operator (A3D), which uses different weights to fuse 3D context from different 2D slices. Notably, A3D is NOT translation-equivariant while it significantly outperforms existing symmetric context fusion operators without introducing large computational overhead. We validate the effectiveness of the proposed method by extensive experiments on DeepLesion benchmark, a large-scale public dataset for universal lesion detection from computed tomography (CT). The proposed A3D consistently outperforms symmetric context fusion operators by considerable margins, and establishes a new state of the art on DeepLesion. To facilitate open research, our code and model in PyTorch are available at https://github.com/M3DV/AlignShift.