# Resource-Efficient Multiview Perception: Integrating Semantic Masking with Masked Autoencoders

arXiv: https://arxiv.org/abs/2410.04817 · HTML: https://mlsanity.com/paper/2410.04817/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 49 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Builds on (most cited first, 30 of 31)

- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Object Detection in 20 Years: A Survey](https://mlsanity.com/paper/1905.05055/) (2019)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [Deep Learning Enabled Semantic Communication Systems](https://mlsanity.com/paper/2006.10685/) (2020)
- [Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps](https://mlsanity.com/paper/2209.12836/) (2022)
- [An Introduction to Deep Learning for the Physical Layer](https://mlsanity.com/paper/1702.00832/) (2017)
- [Multiple Object Tracking: A Literature Review](https://mlsanity.com/paper/1409.7618/) (2014)
- [SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders](https://mlsanity.com/paper/2206.10207/) (2022)
- [DeepJSCC-f: Deep Joint Source-Channel Coding of Images with Feedback](https://mlsanity.com/paper/1911.11174/) (2019)
- [Features for Multi-Target Multi-Camera Tracking and Re-Identification](https://mlsanity.com/paper/1803.10859/) (2018)
- [Task-Oriented Multi-User Semantic Communications](https://mlsanity.com/paper/2112.10255/) (2021)
- [A Lite Distributed Semantic Communication System for Internet of Things](https://mlsanity.com/paper/2007.11095/) (2020)
- [Multiview Detection with Feature Perspective Transformation](https://mlsanity.com/paper/2007.07247/) (2020)
- [Less Data, More Knowledge: Building Next Generation Semantic Communication Networks](https://mlsanity.com/paper/2211.14343/) (2022)
- [Robust Semantic Communications with Masked VQ-VAE Enabled Codebook](https://mlsanity.com/paper/2206.04011/) (2022)
- [Multiview Detection with Shadow Transformer (and View-Coherent Data Augmentation)](https://mlsanity.com/paper/2108.05888/) (2021)
- [LMGP: Lifted Multicut Meets Geometry Projections for Multi-Camera Multi-Object Tracking](https://mlsanity.com/paper/2111.11892/) (2021)
- [EarlyBird: Early-Fusion for Multi-View Tracking in the Bird's Eye View](https://mlsanity.com/paper/2310.13350/) (2023)
- [Task-Oriented Image Transmission for Scene Classification in Unmanned Aerial Systems](https://mlsanity.com/paper/2112.10948/) (2021)
- [Lifting Multi-View Detection and Tracking to the Bird's Eye View](https://mlsanity.com/paper/2403.12573/) (2024)
- [Semantic Communications for Speech Signals](https://mlsanity.com/paper/2012.05369/) (2020)
- [Multi-Drone based Single Object Tracking with Agent Sharing Network](https://mlsanity.com/paper/2003.06994/) (2020)
- [Wireless End-to-End Image Transmission System using Semantic Communications](https://mlsanity.com/paper/2302.13721/) (2023)
- [Artificial Intelligence Techniques for Next-Generation Mega Satellite Networks](https://mlsanity.com/paper/2207.00414/) (2022)
- [Communication-Efficient Framework for Distributed Image Semantic Wireless Transmission](https://mlsanity.com/paper/2308.03713/) (2023)
- [DiffPMAE: Diffusion Masked Autoencoders for Point Cloud Reconstruction](https://mlsanity.com/paper/2312.03298/) (2023)

## Built on by (most cited first, 2 of 2)

- [MVTrajecter: Multi-View Pedestrian Tracking with Trajectory Motion Cost and Trajectory Appearance Cost](https://mlsanity.com/paper/2509.01157/) (2025)
- [Multi-view Crowd Tracking Transformer with View-Ground Interactions Under Large Real-World Scenes](https://mlsanity.com/paper/2604.19318/) (2026)

## Abstract (arXiv)

Multiview systems have become a key technology in modern computer vision, offering advanced capabilities in scene understanding and analysis. However, these systems face critical challenges in bandwidth limitations and computational constraints, particularly for resource-limited camera nodes like drones. This paper presents a novel approach for communication-efficient distributed multiview detection and tracking using masked autoencoders (MAEs). We introduce a semantic-guided masking strategy that leverages pre-trained segmentation models and a tunable power function to prioritize informative image regions. This approach, combined with an MAE, reduces communication overhead while preserving essential visual information. We evaluate our method on both virtual and real-world multiview datasets, demonstrating comparable performance in terms of detection and tracking performance metrics compared to state-of-the-art techniques, even at high masking ratios. Our selective masking algorithm outperforms random masking, maintaining higher accuracy and precision as the masking ratio increases. Furthermore, our approach achieves a significant reduction in transmission data volume compared to baseline methods, thereby balancing multiview tracking performance with communication efficiency.