# Choosing Smartly: Adaptive Multimodal Fusion for Object Detection in Changing Environments

arXiv: https://arxiv.org/abs/1707.05733 · HTML: https://mlsanity.com/paper/1707.05733/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2017.

## Built on by (most cited first, 30 of 34)

- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [The Ingredients for Robotic Diffusion Transformers](https://mlsanity.com/paper/2410.10088/) (2024)
- [Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learning](https://mlsanity.com/paper/2412.12953/) (2024)
- [Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding](https://mlsanity.com/paper/2501.04693/) (2025)
- [Improving Multi-Modal Learning with Uni-Modal Teachers](https://mlsanity.com/paper/2106.11059/) (2021)
- [Predicting the Future from First Person (Egocentric) Vision: A Survey](https://mlsanity.com/paper/2107.13411/) (2021)
- [Defending Multimodal Fusion Models against Single-Source Adversaries](https://mlsanity.com/paper/2206.12714/) (2022)
- [Ultralytics YOLO Evolution: An Overview of YOLO27, YOLO26, YOLO11, YOLOv8, and YOLOv5 Object Detectors for Computer Vision and Pattern Recognition](https://mlsanity.com/paper/2510.09653/) (2025)
- [When Pedestrian Detection Meets Multi-Modal Learning: Generalist Model and Benchmark Dataset](https://mlsanity.com/paper/2407.10125/) (2024)
- [Cocoon: Robust Multi-Modal Perception with Uncertainty-Aware Sensor Fusion](https://mlsanity.com/paper/2410.12592/) (2024)
- [Robots Autonomously Detecting People: A Multimodal Deep Contrastive Learning Method Robust to Intraclass Variations](https://mlsanity.com/paper/2203.00187/) (2022)
- [An Unsupervised Domain Adaptive Approach for Multimodal 2D Object Detection in Adverse Weather Conditions](https://mlsanity.com/paper/2203.03568/) (2022)
- [Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in Autonomous Driving](https://mlsanity.com/paper/2307.16121/) (2023)
- [Exploring Model Learning Heterogeneity for Boosting Ensemble Robustness](https://mlsanity.com/paper/2310.02237/) (2023)
- [PedDet: Adaptive Spectral Optimization for Multimodal Pedestrian Detection](https://mlsanity.com/paper/2502.14063/) (2025)
- [Pay "Attention" to Adverse Weather: Weather-aware Attention-based Object Detection](https://mlsanity.com/paper/2204.10803/) (2022)
- [A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusion](https://mlsanity.com/paper/2501.07451/) (2025)
- [Camera-Radar Perception for Autonomous Vehicles and ADAS: Concepts, Datasets and Metrics](https://mlsanity.com/paper/2303.04302/) (2023)
- [Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers](https://mlsanity.com/paper/2509.05086/) (2025)
- [Semi-MoE: Mixture-of-Experts meets Semi-Supervised Histopathology Segmentation](https://mlsanity.com/paper/2509.13834/) (2025)
- [DBF: Dynamic Belief Fusion for Combining Multiple Object Detectors](https://mlsanity.com/paper/2204.02890/) (2022)
- [LDTrack: Dynamic People Tracking by Service Robots using Diffusion Models](https://mlsanity.com/paper/2402.08774/) (2024)
- [AI perspectives in Smart Cities and Communities to enable road vehicle automation and smart traffic control](https://mlsanity.com/paper/2104.03150/) (2021)
- [TinyM2Net: A Flexible System Algorithm Co-designed Multimodal Learning Framework for Tiny Devices](https://mlsanity.com/paper/2202.04303/) (2022)
- [Training-Free Robust Multimodal Learning via Sample-Wise Jacobian Regularization](https://mlsanity.com/paper/2204.02485/) (2022)
- [Informative Data Selection with Uncertainty for Multi-modal Object Detection](https://mlsanity.com/paper/2304.11697/) (2023)
- [SAGE: Shape-Adapting Gated Experts for Adaptive Histopathology Image Segmentation](https://mlsanity.com/paper/2511.18493/) (2025)
- [Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentation](https://mlsanity.com/paper/2604.13761/) (2026)
- [Detecting Humans in RGB-D Data with CNNs](https://mlsanity.com/paper/2207.08064/) (2022)

## Abstract (arXiv)

Object detection is an essential task for autonomous robots operating in dynamic and changing environments. A robot should be able to detect objects in the presence of sensor noise that can be induced by changing lighting conditions for cameras and false depth readings for range sensors, especially RGB-D cameras. To tackle these challenges, we propose a novel adaptive fusion approach for object detection that learns weighting the predictions of different sensor modalities in an online manner. Our approach is based on a mixture of convolutional neural network (CNN) experts and incorporates multiple modalities including appearance, depth and motion. We test our method in extensive robot experiments, in which we detect people in a combined indoor and outdoor scenario from RGB-D data, and we demonstrate that our method can adapt to harsh lighting changes and severe camera motion blur. Furthermore, we present a new RGB-D dataset for people detection in mixed in- and outdoor environments, recorded with a mobile robot. Code, pretrained models and dataset are available at http://adaptivefusion.cs.uni-freiburg.de