# Deep Spatiotemporal Models for Robust Proprioceptive Terrain Classification

arXiv: https://arxiv.org/abs/1804.00736 · HTML: https://mlsanity.com/paper/1804.00736/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2018.

## Built on by (most cited first, 7 of 7)

- [Catch Me If You Hear Me: Audio-Visual Navigation in Complex Unmapped Environments with Moving Sounds](https://mlsanity.com/paper/2111.14843/) (2021)
- [Learning Visual Locomotion with Cross-Modal Supervision](https://mlsanity.com/paper/2211.03785/) (2022)
- [Pay Self-Attention to Audio-Visual Navigation](https://mlsanity.com/paper/2210.01353/) (2022)
- [RCA: Ride Comfort-Aware Visual Navigation via Self-Supervised Learning](https://mlsanity.com/paper/2207.14460/) (2022)
- [Classical and Quantum Physical Reservoir Computing for Onboard Artificial Intelligence Systems: A Perspective](https://mlsanity.com/paper/2407.04717/) (2024)
- [Proprioception Is All You Need: Terrain Classification for Boreal Forests](https://mlsanity.com/paper/2403.16877/) (2024)
- [TrackletMapper: Ground Surface Segmentation and Mapping from Traffic Participant Trajectories](https://mlsanity.com/paper/2209.05247/) (2022)

## Abstract (arXiv)

Terrain classification is a critical component of any autonomous mobile robot system operating in unknown real-world environments. Over the years, several proprioceptive terrain classification techniques have been introduced to increase robustness or act as a fallback for traditional vision based approaches. However, they lack widespread adaptation due to various factors that include inadequate accuracy, robustness and slow run-times. In this paper, we use vehicle-terrain interaction sounds as a proprioceptive modality and propose a deep Long-Short Term Memory (LSTM) based recurrent model that captures both the spatial and temporal dynamics of such a problem, thereby overcoming these past limitations. Our model consists of a new Convolution Neural Network (CNN) architecture that learns deep spatial features, complemented with LSTM units that learn complex temporal dynamics. Experiments on two extensive datasets collected with different microphones on various indoor and outdoor terrains demonstrate state-of-the-art performance compared to existing techniques. We additionally evaluate the performance in adverse acoustic conditions with high ambient noise and propose a noise-aware training scheme that enables learning of more generalizable models that are essential for robust real-world deployments.