VibeAct: Vibration to Actions for Contact-Rich Reactive Robot Dexterity

Dexterous manipulation depends on contact events that are fast, local, and often visually occluded. Piezoelectric microphones provide compact, high-bandwidth sensing of these interactions, but the resulting vibro-acoustic signals are difficult to simulate faithfully for end-to-end sim-to-real policy learning on dexterous robot hands. We propose VibeAct, a framework that bridges real vibrotactile sensing and simulation-based reinforcement learning through a shared compact tactile representation. Guided by the microphones' sensitivity to contact transients and sliding-induced vibrations, we define this representation as binary contact onset, binary slip presence, and scalar slip magnitude for each fingertip. In the real world, we embed piezoelectric microphones in a dexterous robot hand and collect vibro-acoustic signals synchronized with robot and object states during teleoperation. We replay the recorded states in a calibrated digital clone to generate per-finger contact and slip labels for training a tactile estimator to infer the tactile representation from real microphone signals. In simulation, manipulation policies are trained using the same representation computed directly from simulated contact dynamics. This decoupling lets policies exploit rapid tactile feedback without simulating raw audio. Across eight contact-rich manipulation tasks, VibeAct improves average success from 21.6% to 50.5% over a baseline using proprioception and point clouds in simulation. The transferred policies also outperform the baseline on all three tasks evaluated on a physical dexterous hand-arm platform. Project videos and additional details are at https://vibeact.github.io/.