# Ensemble Learning for Fusion of Multiview Vision with Occlusion and Missing Information: Framework and Evaluations with Real-World Data and Applications in Driver Hand Activity Recognition

arXiv: https://arxiv.org/abs/2301.12592 · HTML: https://mlsanity.com/paper/2301.12592/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 79 references that are in the map.
- First posted on arXiv 30 Jan 2023.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [PointPainting: Sequential Fusion for 3D Object Detection](https://mlsanity.com/paper/1911.10150/) (2019)
- [DeepPose: Human Pose Estimation via Deep Neural Networks](https://mlsanity.com/paper/1312.4659/) (2013)
- [Deep Continuous Fusion for Multi-Sensor 3D Object Detection](https://mlsanity.com/paper/2012.10992/) (2020)
- [FractalNet: Ultra-Deep Neural Networks without Residuals](https://mlsanity.com/paper/1605.07648/) (2016)
- [Deep Learning-Based Human Pose Estimation: A Survey](https://mlsanity.com/paper/2012.13392/) (2020)
- [End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds](https://mlsanity.com/paper/1910.06528/) (2019)
- [Domain Adaptive Ensemble Learning](https://mlsanity.com/paper/2003.07325/) (2020)
- [Learning Feature Pyramids for Human Pose Estimation](https://mlsanity.com/paper/1708.01101/) (2017)
- [Dynamic Multimodal Fusion](https://mlsanity.com/paper/2204.00102/) (2022)
- [MultiViz: Towards Visualizing and Understanding Multimodal Models](https://mlsanity.com/paper/2207.00056/) (2022)
- [Driver Distraction Identification with an Ensemble of Convolutional Neural Networks](https://mlsanity.com/paper/1901.09097/) (2019)
- [2D Human Pose Estimation: A Survey](https://mlsanity.com/paper/2204.07370/) (2022)
- [A Comparative Analysis of Decision-Level Fusion for Multimodal Driver Behaviour Understanding](https://mlsanity.com/paper/2204.04734/) (2022)
- [Looking at the Driver/Rider in Autonomous Vehicles to Predict Take-Over Readiness](https://mlsanity.com/paper/1811.06047/) (2018)
- [Autonomous Vehicles that Alert Humans to Take-Over Controls: Modeling with Real-World Data](https://mlsanity.com/paper/2104.11489/) (2021)
- [Safe Control Transitions: Machine Vision Based Observable Readiness Index and Data-Driven Takeover Time Prediction](https://mlsanity.com/paper/2301.05805/) (2023)
- [Predicting Take-over Time for Autonomous Driving with Real-World Data: Robust Data Augmentation, Models, and Evaluation](https://mlsanity.com/paper/2107.12932/) (2021)
- [Looking at Hands in Autonomous Vehicles: A ConvNet Approach using Part Affinity Fields](https://mlsanity.com/paper/1804.01176/) (2018)
- [HandyNet: A One-stop Solution to Detect, Segment, Localize & Analyze Driver Hands](https://mlsanity.com/paper/1804.07834/) (2018)
- [Subject Cross Validation in Human Activity Recognition](https://mlsanity.com/paper/1904.02666/) (2019)

## Built on by (most cited first, 4 of 4)

- [doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigation](https://mlsanity.com/paper/2412.05893/) (2024)
- [Can Vision-Language Models Understand and Interpret Dynamic Gestures from Pedestrians? Pilot Datasets and Exploration Towards Instructive Nonverbal Commands for Cooperative Autonomous Vehicles](https://mlsanity.com/paper/2504.10873/) (2025)
- [Evaluating Vision-Language Models for Zero-Shot Detection, Classification, and Association of Motorcycles, Passengers, and Helmets](https://mlsanity.com/paper/2408.02244/) (2024)
- [Driver Activity Classification Using Generalizable Representations from Vision-Language Models](https://mlsanity.com/paper/2404.14906/) (2024)

## Abstract (arXiv)

Multi-sensor frameworks provide opportunities for ensemble learning and sensor fusion to make use of redundancy and supplemental information, helpful in real-world safety applications such as continuous driver state monitoring which necessitate predictions even in cases where information may be intermittently missing. We define this problem of intermittent instances of missing information (by occlusion, noise, or sensor failure) and design a learning framework around these data gaps, proposing and analyzing an imputation scheme to handle missing information. We apply these ideas to tasks in camera-based hand activity classification for robust safety during autonomous driving. We show that a late-fusion approach between parallel convolutional neural networks can outperform even the best-placed single camera model in estimating the hands' held objects and positions when validated on within-group subjects, and that our multi-camera framework performs best on average in cross-group validation, and that the fusion approach outperforms ensemble weighted majority and model combination schemes.