# GATE3D: Generalized Attention-based Task-synergized Estimation in 3D*

arXiv: https://arxiv.org/abs/2504.11014 · HTML: https://mlsanity.com/paper/2504.11014/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 41 references that are in the map.
- First posted on arXiv 15 Apr 2025.

## Builds on (most cited first, 30 of 30)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [Categorical Depth Distribution Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2103.01100/) (2021)
- [3D Bounding Box Estimation Using Deep Learning and Geometry](https://mlsanity.com/paper/1612.00496/) (2016)
- [M3D-RPN: Monocular 3D Region Proposal Network for Object Detection](https://mlsanity.com/paper/1907.06038/) (2019)
- [MOTS: Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/1902.03604/) (2019)
- [SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation](https://mlsanity.com/paper/2002.10111/) (2020)
- [Objects are Different: Flexible Monocular 3D Object Detection](https://mlsanity.com/paper/2104.02323/) (2021)
- [Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild](https://mlsanity.com/paper/2207.10660/) (2022)
- [Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations](https://mlsanity.com/paper/2012.09988/) (2020)
- [MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships](https://mlsanity.com/paper/2003.00504/) (2020)
- [ROI-10D: Monocular Lifting of 2D Detection to 6D Pose and Metric Shape](https://mlsanity.com/paper/1812.02781/) (2018)
- [MonoDTR: Monocular 3D Object Detection with Depth-Aware Transformer](https://mlsanity.com/paper/2203.10981/) (2022)
- [FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detection](https://mlsanity.com/paper/2106.12449/) (2021)
- [Virtual Sparse Convolution for Multimodal 3D Object Detection](https://mlsanity.com/paper/2303.02314/) (2023)
- [Combo Loss: Handling Input and Output Imbalance in Multi-Organ Segmentation](https://mlsanity.com/paper/1805.02798/) (2018)
- [Learning Auxiliary Monocular Contexts Helps Monocular 3D Object Detection](https://mlsanity.com/paper/2112.04628/) (2021)
- [MonoCD: Monocular 3D Object Detection with Complementary Depths](https://mlsanity.com/paper/2404.03181/) (2024)
- [MonoDGP: Monocular 3D Object Detection with Decoupled-Query and Geometry-Error Priors](https://mlsanity.com/paper/2410.19590/) (2024)
- [SEED: A Simple and Effective 3D DETR in Point Clouds](https://mlsanity.com/paper/2407.10749/) (2024)
- [GAFusion: Adaptive Fusing LiDAR and Camera with Multiple Guidance for 3D Object Detection](https://mlsanity.com/paper/2411.00340/) (2024)
- [Cross-Modality 3D Object Detection](https://mlsanity.com/paper/2008.10436/) (2020)
- [CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification](https://mlsanity.com/paper/2203.06760/) (2022)
- [Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data](https://mlsanity.com/paper/2411.15657/) (2024)
- [HeightFormer: A Semantic Alignment Monocular 3D Object Detection Method from Roadside Perspective](https://mlsanity.com/paper/2410.07758/) (2024)

## Built on by (most cited first, 4 of 4)

- [IDEAL-M3D: Instance Diversity-Enriched Active Learning for Monocular 3D Detection](https://mlsanity.com/paper/2511.19301/) (2025)
- [LeAD-M3D: Leveraging Asymmetric Distillation for Real-Time Monocular 3D Detection](https://mlsanity.com/paper/2512.05663/) (2025)
- [Contour-Guided Query-Based Feature Fusion for Boundary-Aware and Generalizable Cardiac Ultrasound Segmentation](https://mlsanity.com/paper/2603.28110/) (2026)
- [MAGneT-3D: Monocular and Domain-Generalizable Temporal 3D Detection](https://mlsanity.com/paper/2608.14282/) (2026)

## Abstract (arXiv)

The emerging trend in computer vision emphasizes developing universal models capable of simultaneously addressing multiple diverse tasks. Such universality typically requires joint training across multi-domain datasets to ensure effective generalization. However, monocular 3D object detection presents unique challenges in multi-domain training due to the scarcity of datasets annotated with accurate 3D ground-truth labels, especially beyond typical road-based autonomous driving contexts. To address this challenge, we introduce a novel weakly supervised framework leveraging pseudo-labels. Current pretrained models often struggle to accurately detect pedestrians in non-road environments due to inherent dataset biases. Unlike generalized image-based 2D object detection models, achieving similar generalization in monocular 3D detection remains largely unexplored. In this paper, we propose GATE3D, a novel framework designed specifically for generalized monocular 3D object detection via weak supervision. GATE3D effectively bridges domain gaps by employing consistency losses between 2D and 3D predictions. Remarkably, our model achieves competitive performance on the KITTI benchmark as well as on an indoor-office dataset collected by us to evaluate the generalization capabilities of our framework. Our results demonstrate that GATE3D significantly accelerates learning from limited annotated data through effective pre-training strategies, highlighting substantial potential for broader impacts in robotics, augmented reality, and virtual reality applications. Project page: https://ies0411.github.io/GATE3D/