# Visual-Tactile Cross-Modal Data Generation using Residue-Fusion GAN with Feature-Matching and Perceptual Losses

arXiv: https://arxiv.org/abs/2107.05468 · HTML: https://mlsanity.com/paper/2107.05468/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 62 references that are in the map.
- First posted on arXiv 12 Jul 2021.

## Builds on (most cited first, 29 of 29)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Pros and Cons of GAN Evaluation Measures](https://mlsanity.com/paper/1802.03446/) (2018)
- [Visually Indicated Sounds](https://mlsanity.com/paper/1512.08512/) (2015)
- [Revisiting Active Perception](https://mlsanity.com/paper/1603.02729/) (2016)
- [The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?](https://mlsanity.com/paper/1710.05512/) (2017)
- [Connecting Touch and Vision via Cross-Modal Prediction](https://mlsanity.com/paper/1906.06322/) (2019)
- [Deep Cross-Modal Audio-Visual Generation](https://mlsanity.com/paper/1704.08292/) (2017)
- [Safe Robot Navigation via Multi-Modal Anomaly Detection](https://mlsanity.com/paper/2001.07934/) (2020)
- [Towards Audio to Scene Image Synthesis using Generative Adversarial Network](https://mlsanity.com/paper/1808.04108/) (2018)
- [Connecting Look and Feel: Associating the visual and tactile properties of physical materials](https://mlsanity.com/paper/1704.03822/) (2017)
- [Shape-independent Hardness Estimation Using Deep Learning and a GelSight Tactile Sensor](https://mlsanity.com/paper/1704.03955/) (2017)
- ["Touching to See" and "Seeing to Feel": Robotic Cross-modal SensoryData Generation for Visual-Tactile Perception](https://mlsanity.com/paper/1902.06273/) (2019)
- [Cascade Attention Guided Residue Learning GAN for Cross-Modal Translation](https://mlsanity.com/paper/1907.01826/) (2019)
- [Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images](https://mlsanity.com/paper/1803.03435/) (2018)
- [On Intra-Class Variance for Deep Learning of Classifiers](https://mlsanity.com/paper/1901.11186/) (2019)
- [Vibrotactile Signal Generation from Texture Images or Attributes using Generative Adversarial Network](https://mlsanity.com/paper/1902.07480/) (2019)
- [Learning an Action-Conditional Model for Haptic Texture Generation](https://mlsanity.com/paper/1909.13025/) (2019)
- [Teaching Cameras to Feel: Estimating Tactile Physical Properties of Surfaces From Images](https://mlsanity.com/paper/2004.14487/) (2020)

## Built on by (most cited first, 6 of 6)

- [Controllable Visual-Tactile Synthesis](https://mlsanity.com/paper/2305.03051/) (2023)
- [Visuo-Tactile Zero-Shot Object Recognition with Vision-Language Model](https://mlsanity.com/paper/2409.09276/) (2024)
- [Multi-Modal World Model for Physical Robot Interactions: Simultaneous Visual and Tactile Predictions for Enhanced Accuracy](https://mlsanity.com/paper/2304.11193/) (2023)
- [Synesthesia of Vehicles: Tactile Data Synthesis from Visual Inputs](https://mlsanity.com/paper/2602.01832/) (2026)
- [MultiDiffSense: Diffusion-Based Multi-Modal Visuo-Tactile Image Generation Conditioned on Object Shape and Contact Pose](https://mlsanity.com/paper/2602.19348/) (2026)
- [The Language of Touch: Translating Vibrations into Text with Dual-Branch Learning](https://mlsanity.com/paper/2603.26804/) (2026)

## Abstract (arXiv)

Existing psychophysical studies have revealed that the cross-modal visual-tactile perception is common for humans performing daily activities. However, it is still challenging to build the algorithmic mapping from one modality space to another, namely the cross-modal visual-tactile data translation/generation, which could be potentially important for robotic operation. In this paper, we propose a deep-learning-based approach for cross-modal visual-tactile data generation by leveraging the framework of the generative adversarial networks (GANs). Our approach takes the visual image of a material surface as the visual data, and the accelerometer signal induced by the pen-sliding movement on the surface as the tactile data. We adopt the conditional-GAN (cGAN) structure together with the residue-fusion (RF) module, and train the model with the additional feature-matching (FM) and perceptual losses to achieve the cross-modal data generation. The experimental results show that the inclusion of the RF module, and the FM and the perceptual losses significantly improves cross-modal data generation performance in terms of the classification accuracy upon the generated data and the visual similarity between the ground-truth and the generated data.