# Learning to Read Braille: Bridging the Tactile Reality Gap with Diffusion Models

arXiv: https://arxiv.org/abs/2304.01182 · HTML: https://mlsanity.com/paper/2304.01182/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 34 of its 43 references that are in the map.
- First posted on arXiv 3 Apr 2023.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [Tackling the Generative Learning Trilemma with Denoising Diffusion GANs](https://mlsanity.com/paper/2112.07804/) (2021)
- [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020)
- [Pretraining is All You Need for Image-to-Image Translation](https://mlsanity.com/paper/2205.12952/) (2022)
- [AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners](https://mlsanity.com/paper/2302.01877/) (2023)
- [DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics](https://mlsanity.com/paper/2210.02438/) (2022)
- [TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensors](https://mlsanity.com/paper/2012.08456/) (2020)
- [GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Finger](https://mlsanity.com/paper/1803.00628/) (2018)
- [Tactile-RL for Insertion: Generalization to Objects of Unknown Geometry](https://mlsanity.com/paper/2104.01167/) (2021)
- [Cable Manipulation with a Tactile-Reactive Gripper](https://mlsanity.com/paper/1910.02860/) (2019)
- [Taxim: An Example-based Simulation Model for GelSight Tactile Sensors](https://mlsanity.com/paper/2109.04027/) (2021)
- [Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model](https://mlsanity.com/paper/2208.12675/) (2022)
- [MidasTouch: Monte-Carlo inference over distributions across sliding touch](https://mlsanity.com/paper/2210.14210/) (2022)
- [Active Clothing Material Perception using Tactile Sensing and Deep Learning](https://mlsanity.com/paper/1711.00574/) (2017)
- [OmniTact: A Multi-Directional High Resolution Touch Sensor](https://mlsanity.com/paper/2003.06965/) (2020)
- [Tactile Object Pose Estimation from the First Touch with Geometric Contact Rendering](https://mlsanity.com/paper/2012.05205/) (2020)
- [Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation](https://mlsanity.com/paper/2106.08796/) (2021)
- [Tactile Gym 2.0: Sim-to-real Deep Reinforcement Learning for Comparing Low-cost High-Resolution Robot Touch](https://mlsanity.com/paper/2207.10763/) (2022)

## Built on by (most cited first, 18 of 18)

- [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024)
- [Binding Touch to Everything: Learning Unified Multimodal Tactile Representations](https://mlsanity.com/paper/2401.18084/) (2024)
- [A Touch, Vision, and Language Dataset for Multimodal Alignment](https://mlsanity.com/paper/2402.13232/) (2024)
- [Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations](https://mlsanity.com/paper/2408.04380/) (2024)
- [Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation](https://mlsanity.com/paper/2312.13469/) (2023)
- [Generating Visual Scenes from Touch](https://mlsanity.com/paper/2309.15117/) (2023)
- [Tactile-Augmented Radiance Fields](https://mlsanity.com/paper/2405.04534/) (2024)
- [Composable Part-Based Manipulation](https://mlsanity.com/paper/2405.05876/) (2024)
- [TouchSDF: A DeepSDF Approach for 3D Shape Reconstruction using Vision-Based Tactile Sensing](https://mlsanity.com/paper/2311.12602/) (2023)
- [Sim2Real Manipulation on Unknown Objects with Tactile-based Reinforcement Learning](https://mlsanity.com/paper/2403.12170/) (2024)
- [Sim2Real Bilevel Adaptation for Object Surface Classification using Vision-Based Tactile Sensors](https://mlsanity.com/paper/2311.01380/) (2023)
- [DIMA: DIffusing Motion Artifacts for unsupervised correction in brain MRI images](https://mlsanity.com/paper/2504.06767/) (2025)
- [Assessing the use of Diffusion models for motion artifact correction in brain MRI](https://mlsanity.com/paper/2502.01418/) (2025)
- [Cross-Sensor Touch Generation](https://mlsanity.com/paper/2510.09817/) (2025)
- [Learning human-to-robot handovers through 3D scene reconstruction](https://mlsanity.com/paper/2507.08726/) (2025)
- [Survey on Controlable Image Synthesis with Deep Learning](https://mlsanity.com/paper/2307.10275/) (2023)
- [ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generation](https://mlsanity.com/paper/2505.20498/) (2025)
- [MultiDiffSense: Diffusion-Based Multi-Modal Visuo-Tactile Image Generation Conditioned on Object Shape and Contact Pose](https://mlsanity.com/paper/2602.19348/) (2026)

## Abstract (arXiv)

Simulating vision-based tactile sensors enables learning models for contact-rich tasks when collecting real world data at scale can be prohibitive. However, modeling the optical response of the gel deformation as well as incorporating the dynamics of the contact makes sim2real challenging. Prior works have explored data augmentation, fine-tuning, or learning generative models to reduce the sim2real gap. In this work, we present the first method to leverage probabilistic diffusion models for capturing complex illumination changes from gel deformations. Our tactile diffusion model is able to generate realistic tactile images from simulated contact depth bridging the reality gap for vision-based tactile sensing. On real braille reading task with a DIGIT sensor, a classifier trained with our diffusion model achieves 75.74% accuracy outperforming classifiers trained with simulation and other approaches. Project page: https://github.com/carolinahiguera/Tactile-Diffusion