# VisTaNet: Attention Guided Deep Fusion for Surface Roughness Classification

arXiv: https://arxiv.org/abs/2209.08516 · HTML: https://mlsanity.com/paper/2209.08516/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 38 references that are in the map.
- First posted on arXiv 18 Sep 2022.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Wavelet Convolutional Neural Networks](https://mlsanity.com/paper/1805.08620/) (2018)
- [ViTac: Feature Sharing between Vision and Tactile Sensing for Cloth Texture Recognition](https://mlsanity.com/paper/1802.07490/) (2018)
- [Wavelet Convolutional Neural Networks for Texture Classification](https://mlsanity.com/paper/1707.07394/) (2017)
- [Using Filter Banks in Convolutional Neural Networks for Texture Classification](https://mlsanity.com/paper/1601.02919/) (2016)
- [Towards Multidimensional Textural Perception and Classification Through Whisker](https://mlsanity.com/paper/2209.03750/) (2022)

## Abstract (arXiv)

Human texture perception is a weighted average of multi-sensory inputs: visual and tactile. While the visual sensing mechanism extracts global features, the tactile mechanism complements it by extracting local features. The lack of coupled visuotactile datasets in the literature is a challenge for studying multimodal fusion strategies analogous to human texture perception. This paper presents a visual dataset that augments an existing tactile dataset. We propose a novel deep fusion architecture that fuses visual and tactile data using four types of fusion strategies: summation, concatenation, max-pooling, and attention. Our model shows significant performance improvements (97.22%) in surface roughness classification accuracy over tactile only (SVM - 92.60%) and visual only (FENet-50 - 85.01%) architectures. Among the several fusion techniques, attention-guided architecture results in better classification accuracy. Our study shows that analogous to human texture perception, the proposed model chooses a weighted combination of the two modalities (visual and tactile), thus resulting in higher surface roughness classification accuracy; and it chooses to maximize the weightage of the tactile modality where the visual modality fails and vice-versa.