# MultiDiffSense: Diffusion-Based Multi-Modal Visuo-Tactile Image Generation Conditioned on Object Shape and Contact Pose

arXiv: https://arxiv.org/abs/2602.19348 · HTML: https://mlsanity.com/paper/2602.19348/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 22 Feb 2026.

## Builds on (most cited first, 27 of 27)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [DisCo: Disentangled Control for Realistic Human Dance Generation](https://mlsanity.com/paper/2307.00040/) (2023)
- [Touch and Go: Learning from Human-Collected Vision and Touch](https://mlsanity.com/paper/2211.12498/) (2022)
- [ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer](https://mlsanity.com/paper/2204.02389/) (2022)
- [Connecting Touch and Vision via Cross-Modal Prediction](https://mlsanity.com/paper/1906.06322/) (2019)
- [TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensors](https://mlsanity.com/paper/2012.08456/) (2020)
- [Taxim: An Example-based Simulation Model for GelSight Tactile Sensors](https://mlsanity.com/paper/2109.04027/) (2021)
- [ViTac: Feature Sharing between Vision and Tactile Sensing for Cloth Texture Recognition](https://mlsanity.com/paper/1802.07490/) (2018)
- [Generating Visual Scenes from Touch](https://mlsanity.com/paper/2309.15117/) (2023)
- [Learning to Read Braille: Bridging the Tactile Reality Gap with Diffusion Models](https://mlsanity.com/paper/2304.01182/) (2023)
- ["Touching to See" and "Seeing to Feel": Robotic Cross-modal SensoryData Generation for Visual-Tactile Perception](https://mlsanity.com/paper/1902.06273/) (2019)
- [Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching](https://mlsanity.com/paper/1903.03591/) (2019)
- [Simulation of Vision-based Tactile Sensors using Physics based Rendering](https://mlsanity.com/paper/2012.13184/) (2020)
- [Soft Biomimetic Optical Tactile Sensing with the TacTip: A Review](https://mlsanity.com/paper/2105.14455/) (2021)
- [Pose-Based Tactile Servoing: Controlled Soft Touch using Deep Learning](https://mlsanity.com/paper/2012.02504/) (2020)
- [Visual-Tactile Cross-Modal Data Generation using Residue-Fusion GAN with Feature-Matching and Perceptual Losses](https://mlsanity.com/paper/2107.05468/) (2021)
- [ViTacTip: Design and Verification of a Novel Biomimetic Physical Vision-Tactile Fusion Sensor](https://mlsanity.com/paper/2402.00199/) (2024)
- [Deep Tactile Experience: Estimating Tactile Sensor Output from Depth Sensor Data](https://mlsanity.com/paper/2110.08946/) (2021)
- [TacMMs: Tactile Mobile Manipulators for Warehouse Automation](https://mlsanity.com/paper/2306.16783/) (2023)
- [MagicTac: A Novel High-Resolution 3D Multi-layer Grid-Based Tactile Sensor](https://mlsanity.com/paper/2402.01366/) (2024)
- [TacPalm: A Soft Gripper with a Biomimetic Optical Tactile Palm for Stable Precise Grasping](https://mlsanity.com/paper/2409.15239/) (2024)
- [Design and Benchmarking of A Multi-Modality Sensor for Robotic Manipulation with GAN-Based Cross-Modality Interpretation](https://mlsanity.com/paper/2501.02303/) (2025)
- [Vision-based Tactile Image Generation via Contact Condition-guided Diffusion Model](https://mlsanity.com/paper/2412.01639/) (2024)
- [MagicGripper: A Multimodal Sensor-Integrated Gripper for Contact-Rich Robotic Manipulation](https://mlsanity.com/paper/2505.24382/) (2025)

## Abstract (arXiv)

Acquiring aligned visuo-tactile datasets is slow and costly, requiring specialised hardware and large-scale data collection. Synthetic generation is promising, but prior methods are typically single-modality, limiting cross-modal learning. We present MultiDiffSense, a unified diffusion model that synthesises images for multiple vision-based tactile sensors (ViTac, TacTip, ViTacTip) within a single architecture. Our approach uses dual conditioning on CAD-derived, pose-aligned depth maps and structured prompts that encode sensor type and 4-DoF contact pose, enabling controllable, physically consistent multi-modal synthesis. Evaluating on 8 objects (5 seen, 3 novel) and unseen poses, MultiDiffSense outperforms a Pix2Pix cGAN baseline in SSIM by +36.3% (ViTac), +134.6% (ViTacTip), and +64.7% (TacTip). For downstream 3-DoF pose estimation, mixing 50% synthetic with 50% real halves the required real data while maintaining competitive performance. MultiDiffSense alleviates the data-collection bottleneck in tactile sensing and enables scalable, controllable multi-modal dataset generation for robotic applications.