mlsanityData to 7 Oct 2026

Touch and Go: Learning from Human-Collected Vision and Touch

arXiv 2211.12498 ↗First posted 22 Nov 2022cs.CVcited by 84

Not on the map. Why?

Abstract (arXiv)

The ability to associate touch with sight is essential for tasks that require physically interacting with objects in the world. We propose a dataset with paired visual and tactile data called Touch and Go, in which human data collectors probe objects in natural environments using tactile sensors, while simultaneously recording egocentric video. In contrast to previous efforts, which have largely been confined to lab settings or simulated environments, our dataset spans a large number of "in the wild" objects and scenes. To demonstrate our dataset's effectiveness, we successfully apply it to a variety of tasks: 1) self-supervised visuo-tactile feature learning, 2) tactile-driven image stylization, i.e., making the visual appearance of an object more consistent with a given tactile signal, and 3) predicting future frames of a tactile signal from visuo-tactile inputs.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 50 of its 85 references that are in the map. First posted on arXiv 22 Nov 2022.

Builds on

50 of 85 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  7. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  8. 2019Contrastive Multiview Codingcited by 1,297
  9. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  10. 2016Generative Adversarial Text to Image Synthesiscited by 735
  11. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  12. 2017Look, Listen and Learncited by 452
38 more
  1. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  2. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  3. 2018Image Generation from Scene Graphscited by 324
  4. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  5. 2018Stochastic Video Generation with a Learned Priorcited by 244
  6. 2017Stochastic Variational Video Predictioncited by 234
  7. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 203
  8. 2019Few-shot Video-to-Video Synthesiscited by 196
  9. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 192
  10. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  11. 2020DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulationcited by 164
  12. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 150
  13. 2015Visually Indicated Soundscited by 128
  14. 2021Paint by Wordcited by 107
  15. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 104
  16. 2018More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touchcited by 100
  17. 2019High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networkscited by 88
  18. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 73
  19. 2017Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learningcited by 67
  20. 2021ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representationscited by 67
  21. 2019Connecting Touch and Vision via Cross-Modal Predictioncited by 66
  22. 2017The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?cited by 65
  23. 2020TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensorscited by 59
  24. 2020Future Video Synthesis with Object Motion Predictioncited by 57
  25. 2018Disentangling Propagation and Generation for Video Predictioncited by 44
  26. 2021Sound-Guided Semantic Image Manipulationcited by 35
  27. 2019Manipulation by Feel: Touch-Based Control with Deep Predictive Modelscited by 34
  28. 2015Spatio-temporal video autoencoder with differentiable memorycited by 32
  29. 2020Sound2Sight: Generating Visual Dynamics from Sound and Contextcited by 28
  30. 2022VIRDO: Visio-tactile Implicit Representations of Deformable Objectscited by 25
  31. 2022Learning Visual Styles from Audio-Visual Associationscited by 22
  32. 2017Shape-independent Hardness Estimation Using Deep Learning and a GelSight Tactile Sensorcited by 19
  33. 2018Learning to Grasp Without Seeingcited by 19
  34. 2019Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matchingcited by 16
  35. 2021Comparing Correspondences: Video Prediction with Correspondence-wise Lossescited by 16
  36. 2022MatFormer: A Generative Model for Procedural Materialscited by 12
  37. 2021Dynamic Modeling of Hand-Object Interactions via Tactile Sensingcited by 8
  38. 2019Bounce and Learn: Modeling Scene Dynamics with Real-World Bouncescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 9; 2024: 18; 2025: 25; 2026: 31.

84 papers.

Sort:

By area:

  1. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  2. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  3. 2024Learning Human-to-Humanoid Real-Time Whole-Body Teleoperationcited by 112
  4. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94
  5. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  6. 2024A Touch, Vision, and Language Dataset for Multimodal Alignmentcited by 68
  7. 2024SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priorscited by 53
  8. 2025AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensorscited by 46
  9. 2024Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Taskscited by 44
  10. 2023The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objectscited by 40
  11. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  12. 2024WorDepth: Variational Language Prior for Monocular Depth Estimationcited by 34
38 more
  1. 2023Understanding 3D Object Interaction from a Single Imagecited by 33
  2. 2024Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Trainingcited by 30
  3. 2023ViT-Lens: Towards Omni-modal Representationscited by 29
  4. 2025Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Grippercited by 29
  5. 2023Generating Visual Scenes from Touchcited by 26
  6. 2024Test-Time Adaptation for Depth Completioncited by 26
  7. 2024Tactile-Augmented Radiance Fieldscited by 26
  8. 2024OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of Allcited by 20
  9. 2024CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusioncited by 17
  10. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  11. 2023Boosting Detection in Crowd Analysis via Underutilized Output Featurescited by 14
  12. 2024NeuroBind: Towards Unified Multimodal Representations for Neural Signalscited by 13
  13. 2026CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decodingcited by 13
  14. 2025OPENTOUCH: Bringing Full-Hand Touch to Real-World Interactioncited by 11
  15. 2026AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perceptioncited by 11
  16. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  17. 2023Controllable Visual-Tactile Synthesiscited by 8
  18. 2024Self-Supervised Audio-Visual Soundscape Stylizationcited by 8
  19. 2025Sensor-Invariant Tactile Representationcited by 8
  20. 2023VCISR: Blind Single Image Super-Resolution with Video Compression Synthetic Datacited by 7
  21. 2024M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulationcited by 7
  22. 2025Universal Visuo-Tactile Video Understanding for Embodied Interactioncited by 7
  23. 2025ConViTac: Aligning Visual-Tactile Fusion with Contrastive Representationscited by 7
  24. 2022MRTNet: Multi-Resolution Temporal Network for Video Sentence Groundingcited by 6
  25. 2023Learning to Taste: A Multimodal Wine Datasetcited by 6
  26. 2024Towards Comprehensive Multimodal Perception: Introducing the Touch-Language-Vision Datasetcited by 6
  27. 2024HyperTaxel: Hyper-Resolution for Taxel-Based Tactile Signals Through Contrastive Learningcited by 6
  28. 2025Surformer v1: Transformer-Based Surface Classification Using Tactile and Vision Featurescited by 4
  29. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 4
  30. 2025X-Capture: An Open-Source Portable Device for Multi-Sensory Learningcited by 3
  31. 2025ViTa-Zero: Zero-shot Visuotactile Object 6D Pose Estimationcited by 3
  32. 2025SToLa: Self-Adaptive Touch-Language Framework with Tactile Commonsense Reasoning in Open-Ended Scenarioscited by 3
  33. 2025MultiModal Action Conditioned Video Generationcited by 3
  34. 2025New York Smells: A Large Multimodal Dataset for Olfactioncited by 3
  35. 2026TacCoRL: Integrating Tactile Feedback into VLA via Simulationcited by 3
  36. 2026UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generationcited by 3
  37. 2023GelFlow: Self-supervised Learning of Optical Flow for Vision-Based Tactile Sensor Displacement Measurementcited by 2
  38. 2025ViTaPEs: Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformerscited by 2

50 most cited of 84 All 84 (JSON)