# Multimodal Multi-User Surface Recognition with the Kernel Two-Sample Test

arXiv: https://arxiv.org/abs/2303.04930 · HTML: https://mlsanity.com/paper/2303.04930/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 55 references that are in the map.
- First posted on arXiv 8 Mar 2023.

## Builds on (most cited first, 11 of 11)

- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [Kernel Mean Embedding of Distributions: A Review and Beyond](https://mlsanity.com/paper/1605.09522/) (2016)
- [Beyond Sharing Weights for Deep Domain Adaptation](https://mlsanity.com/paper/1603.06432/) (2016)
- [From BoW to CNN: Two Decades of Texture Representation for Texture Classification](https://mlsanity.com/paper/1801.10324/) (2018)
- [A Kernel Independence Test for Random Processes](https://mlsanity.com/paper/1402.4501/) (2014)
- [Deep Learning for Tactile Understanding From Visual and Haptic Data](https://mlsanity.com/paper/1511.06065/) (2015)
- [A Kernel Two-sample Test for Dynamical Systems](https://mlsanity.com/paper/2004.11098/) (2020)
- [Deep Learning for Surface Material Classification Using Haptic And Visual Information](https://mlsanity.com/paper/1512.06658/) (2015)
- [Recovery Guarantees for Kernel-based Clustering under Non-parametric Mixture Models](https://mlsanity.com/paper/2110.09476/) (2021)

## Abstract (arXiv)

Machine learning and deep learning have been used extensively to classify physical surfaces through images and time-series contact data. However, these methods rely on human expertise and entail the time-consuming processes of data and parameter tuning. To overcome these challenges, we propose an easily implemented framework that can directly handle heterogeneous data sources for classification tasks. Our data-versus-data approach automatically quantifies distinctive differences in distributions in a high-dimensional space via kernel two-sample testing between two sets extracted from multimodal data (e.g., images, sounds, haptic signals). We demonstrate the effectiveness of our technique by benchmarking against expertly engineered classifiers for visual-audio-haptic surface recognition due to the industrial relevance, difficulty, and competitive baselines of this application; ablation studies confirm the utility of key components of our pipeline. As shown in our open-source code, we achieve 97.2% accuracy on a standard multi-user dataset with 108 surface classes, outperforming the state-of-the-art machine-learning algorithm by 6% on a more difficult version of the task. The fact that our classifier obtains this performance with minimal data processing in the standard algorithm setting reinforces the powerful nature of kernel methods for learning to recognize complex patterns.