# TacSushi: Tactile-Grounded World-Action Modeling for Dexterous Sushi Manipulation

arXiv: https://arxiv.org/abs/2609.19613 · HTML: https://mlsanity.com/paper/2609.19613/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 28 of its 34 references that are in the map.
- First posted on arXiv 17 Sep 2026.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)

## Builds on (most cited first, 28 of 28)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control](https://mlsanity.com/paper/1812.00568/) (2018)
- [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020)
- [Cosmos 3: Omnimodal World Models for Physical AI](https://mlsanity.com/paper/2606.02800/) (2026)
- [MediaPipe Hands: On-device Real-time Hand Tracking](https://mlsanity.com/paper/2006.10214/) (2020)
- [Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1810.10191/) (2018)
- [SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/2011.07215/) (2020)
- [PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics](https://mlsanity.com/paper/2104.03311/) (2021)
- [More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch](https://mlsanity.com/paper/1805.11085/) (2018)
- [RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks](https://mlsanity.com/paper/2205.02909/) (2022)
- [Connecting Touch and Vision via Cross-Modal Prediction](https://mlsanity.com/paper/1906.06322/) (2019)
- [ViTacFormer: Learning Cross-Modal Representation for Visuo-Tactile Dexterous Manipulation](https://mlsanity.com/paper/2506.15953/) (2025)
- [Manipulation by Feel: Touch-Based Control with Deep Predictive Models](https://mlsanity.com/paper/1903.04128/) (2019)
- [Towards Robotic Feeding: Role of Haptics in Fork-based Food Manipulation](https://mlsanity.com/paper/1804.08768/) (2018)
- [Action Conditioned Tactile Prediction: case study on slip prediction](https://mlsanity.com/paper/2205.09430/) (2022)
- [TACT: Humanoid Whole-body Contact Manipulation through Deep Imitation Learning with Tactile Modality](https://mlsanity.com/paper/2506.15146/) (2025)
- [Learning Granular Media Avalanche Behavior for Indirectly Manipulating Obstacles on a Granular Slope](https://mlsanity.com/paper/2407.01898/) (2024)
- [Granular Loco-Manipulation: Repositioning Rocks Through Strategic Sand Avalanche](https://mlsanity.com/paper/2505.12934/) (2025)
- [Sashimi-Bot: Autonomous Tri-manual Advanced Manipulation and Cutting of Deformable Objects](https://mlsanity.com/paper/2511.11223/) (2025)

## Abstract (arXiv)

Dexterous food manipulation requires control under deformation, occlusion, and uncertain contact. We present TacSushi, a tactile-grounded, Cosmos3-based world-action policy that learns from recorded future consequences while acting on current observations. The backbone encodes current RGB, language, and hand state, and feature-wise gated fusion incorporates fingertip tactile features into the action representation. During training, a decoder conditioned on demonstrated action chunks predicts logged future visual observations, task progress, relative contact risk, and tactile summaries; this decoder is removed at deployment. Failed trials provide consequence supervision, but their actions are excluded from imitation. We train TacSushi on 340 successful and 50 failed real-robot trials and compare six methods in 600 separate rollouts across three in-distribution tasks and two out-of-distribution ingredient variants. To assess food quality beyond a single geometric threshold, we score terminal outcomes using an anchored visual-quality protocol that equally weights five human ratings and three vision-language-model ratings per rollout. Full TacSushi achieves 68.3% average in-distribution success and 37.5% out-of-distribution success, compared with 36.7%/10.0% without future-consequence supervision and 25.0%/17.5% with direct tactile concatenation in place of gated fusion. These comparisons support complementary benefits of feature-wise gated tactile fusion and training-only predictive supervision.