# Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors

arXiv: https://arxiv.org/abs/2012.00201 · HTML: https://mlsanity.com/paper/2012.00201/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2020.

## Built on by (most cited first, 17 of 17)

- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress](https://mlsanity.com/paper/2410.04640/) (2024)
- [Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation](https://mlsanity.com/paper/2112.05251/) (2021)
- [Dynamic Multimodal Fusion](https://mlsanity.com/paper/2204.00102/) (2022)
- [Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions](https://mlsanity.com/paper/2111.08456/) (2021)
- [Tools Fail: Detecting Silent Errors in Faulty Tools](https://mlsanity.com/paper/2406.19228/) (2024)
- [Resilient Legged Local Navigation: Learning to Traverse with Compromised Perception End-to-End](https://mlsanity.com/paper/2310.03581/) (2023)
- [Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding](https://mlsanity.com/paper/2211.14648/) (2022)
- [Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrations](https://mlsanity.com/paper/2209.07682/) (2022)
- [Multiscale Sensor Fusion and Continuous Control with Neural CDEs](https://mlsanity.com/paper/2203.08715/) (2022)
- [Non-contact Multimodal Indoor Human Monitoring Systems: A Survey](https://mlsanity.com/paper/2312.07601/) (2023)
- [Multi-modal perception for soft robotic interactions using generative models](https://mlsanity.com/paper/2404.04220/) (2024)
- [AugInsert: Learning Robust Visual-Force Policies via Data Augmentation for Object Assembly Tasks](https://mlsanity.com/paper/2410.14968/) (2024)
- [Inertial Hallucinations -- When Wearable Inertial Devices Start Seeing Things](https://mlsanity.com/paper/2207.06789/) (2022)
- [How do Cross-View and Cross-Modal Alignment Affect Representations in Contrastive Learning?](https://mlsanity.com/paper/2211.13309/) (2022)
- [FusionSense: Tri-Stage Near-Sensor Learning for Runtime-Adaptive Multimodal Edge Intelligence](https://mlsanity.com/paper/2605.22868/) (2026)

## Abstract (arXiv)

Using sensor data from multiple modalities presents an opportunity to encode redundant and complementary features that can be useful when one modality is corrupted or noisy. Humans do this everyday, relying on touch and proprioceptive feedback in visually-challenging environments. However, robots might not always know when their sensors are corrupted, as even broken sensors can return valid values. In this work, we introduce the Crossmodal Compensation Model (CCM), which can detect corrupted sensor modalities and compensate for them. CMM is a representation model learned with self-supervision that leverages unimodal reconstruction loss for corruption detection. CCM then discards the corrupted modality and compensates for it with information from the remaining sensors. We show that CCM learns rich state representations that can be used for contact-rich manipulation policies, even when input modalities are corrupted in ways not seen during training time.