# OmniVaT: Single Domain Generalization for Multimodal Visual-Tactile Learning

arXiv: https://arxiv.org/abs/2601.00352 · HTML: https://mlsanity.com/paper/2601.00352/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 52 references that are in the map.
- First posted on arXiv 1 Jan 2026.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [A Fourier-based Framework for Domain Generalization](https://mlsanity.com/paper/2105.11120/) (2021)
- [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020)
- [Binding Touch to Everything: Learning Unified Multimodal Tactile Representations](https://mlsanity.com/paper/2401.18084/) (2024)
- [Touch and Go: Learning from Human-Collected Vision and Touch](https://mlsanity.com/paper/2211.12498/) (2022)
- [Diverse Embedding Expansion Network and Low-Light Cross-Modality Benchmark for Visible-Infrared Person Re-identification](https://mlsanity.com/paper/2303.14481/) (2023)
- [ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer](https://mlsanity.com/paper/2204.02389/) (2022)
- [PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization](https://mlsanity.com/paper/2307.15199/) (2023)
- [VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation](https://mlsanity.com/paper/2505.09577/) (2025)
- [GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Finger](https://mlsanity.com/paper/1803.00628/) (2018)
- [Frequency-aware Feature Fusion for Dense Image Prediction](https://mlsanity.com/paper/2408.12879/) (2024)
- [AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors](https://mlsanity.com/paper/2502.12191/) (2025)
- [Taxim: An Example-based Simulation Model for GelSight Tactile Sensors](https://mlsanity.com/paper/2109.04027/) (2021)
- [Meta-causal Learning for Single Domain Generalization](https://mlsanity.com/paper/2304.03709/) (2023)
- [The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects](https://mlsanity.com/paper/2306.00956/) (2023)
- [MMRL: Multi-Modal Representation Learning for Vision-Language Models](https://mlsanity.com/paper/2503.08497/) (2025)
- [Domain-invariant Feature Exploration for Domain Generalization](https://mlsanity.com/paper/2207.12020/) (2022)
- [Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Training](https://mlsanity.com/paper/2401.12024/) (2024)
- [Embedding Expansion: Augmentation in Embedding Space for Deep Metric Learning](https://mlsanity.com/paper/2003.02546/) (2020)
- [Soft Prompt Generation for Domain Generalization](https://mlsanity.com/paper/2404.19286/) (2024)
- [AdvST: Revisiting Data Augmentations for Single Domain Generalization](https://mlsanity.com/paper/2312.12720/) (2023)
- [Tac3D: A Novel Vision-based Tactile Sensor for Measuring Forces Distribution and Estimating Friction Coefficient Distribution](https://mlsanity.com/paper/2202.06211/) (2022)
- [Logits DeConfusion with CLIP for Few-Shot Learning](https://mlsanity.com/paper/2504.12104/) (2025)
- [Sensor-Invariant Tactile Representation](https://mlsanity.com/paper/2502.19638/) (2025)

## Abstract (arXiv)

Visual-tactile learning (VTL) enables embodied agents to perceive the physical world by integrating visual (VIS) and tactile (TAC) sensors. However, VTL still suffers from modality discrepancies between VIS and TAC images, as well as domain gaps caused by non-standardized tactile sensors and inconsistent data collection procedures. We formulate these challenges as a new task, termed single domain generalization for multimodal VTL (SDG-VTL). In this paper, we propose an OmniVaT framework that, for the first time, successfully addresses this task. On the one hand, OmniVaT integrates a multimodal fractional Fourier adapter (MFFA) to map VIS and TAC embeddings into a unified embedding-frequency space, thereby effectively mitigating the modality gap without multi-domain training data or careful cross-modal fusion strategies. On the other hand, it also incorporates a discrete tree generation (DTG) module that obtains diverse and reliable multimodal fractional representations through a hierarchical tree structure, thereby enhancing its adaptivity to fluctuating domain shifts in unseen domains. Extensive experiments demonstrate the superior cross-domain generalization performance of OmniVaT on the SDG-VTL task.