# FeelAnyForce: Estimating Contact Force Feedback from Tactile Sensation for Vision-Based Tactile Sensors

arXiv: https://arxiv.org/abs/2410.02048 · HTML: https://mlsanity.com/paper/2410.02048/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 28 of 28)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020)
- [The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?](https://mlsanity.com/paper/1710.05512/) (2017)
- [Manipulation by Feel: Touch-Based Control with Deep Predictive Models](https://mlsanity.com/paper/1903.04128/) (2019)
- [MidasTouch: Monte-Carlo inference over distributions across sliding touch](https://mlsanity.com/paper/2210.14210/) (2022)
- [ShapeMap 3-D: Efficient shape mapping through dense touch and vision](https://mlsanity.com/paper/2109.09884/) (2021)
- [Robotic Tactile Perception of Object Properties: A Review](https://mlsanity.com/paper/1711.03810/) (2017)
- [Tactile Dexterity: Manipulation Primitives with Tactile Feedback](https://mlsanity.com/paper/2002.03236/) (2020)
- [Visuotactile Affordances for Cloth Manipulation with Local Control](https://mlsanity.com/paper/2212.05108/) (2022)
- [Dense Tactile Force Distribution Estimation using GelSlim and inverse FEM](https://mlsanity.com/paper/1810.04621/) (2018)
- [9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation](https://mlsanity.com/paper/2308.14277/) (2023)
- [Maintaining Grasps within Slipping Bound by Monitoring Incipient Slip](https://mlsanity.com/paper/1810.13381/) (2018)
- [AcTExplore: Active Tactile Exploration of Unknown Objects](https://mlsanity.com/paper/2310.08745/) (2023)
- [Collision-aware In-hand 6D Object Pose Estimation using Multiple Vision-based Tactile Sensors](https://mlsanity.com/paper/2301.13667/) (2023)
- [Grasp Stability Prediction with Sim-to-Real Transfer from Tactile Sensing](https://mlsanity.com/paper/2208.02885/) (2022)
- [Robust Learning of Tactile Force Estimation through Robot Interaction](https://mlsanity.com/paper/1810.06187/) (2018)
- [What Matters for Active Texture Recognition With Vision-Based Tactile Sensors](https://mlsanity.com/paper/2403.13701/) (2024)
- [Improving Grasp Stability with Rotation Measurement from Tactile Sensing](https://mlsanity.com/paper/2108.00301/) (2021)
- [In-Mouth Robotic Bite Transfer with Visual and Haptic Sensing](https://mlsanity.com/paper/2211.12705/) (2022)
- [Learning Sequential Acquisition Policies for Robot-Assisted Feeding](https://mlsanity.com/paper/2309.05197/) (2023)
- [ViTacTip: Design and Verification of a Novel Biomimetic Physical Vision-Tactile Fusion Sensor](https://mlsanity.com/paper/2402.00199/) (2024)
- [Slip detection for grasp stabilisation with a multi-fingered tactile robot hand](https://mlsanity.com/paper/2010.01928/) (2020)
- [Sim2Real Bilevel Adaptation for Object Surface Classification using Vision-Based Tactile Sensors](https://mlsanity.com/paper/2311.01380/) (2023)
- [3D Force and Contact Estimation for a Soft-Bubble Visuotactile Sensor Using FEM](https://mlsanity.com/paper/2310.11372/) (2023)
- [GradTac: Spatio-Temporal Gradient Based Tactile Sensing](https://mlsanity.com/paper/2203.07290/) (2022)

## Built on by (most cited first, 11 of 11)

- [A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AI](https://mlsanity.com/paper/2505.01458/) (2025)
- [FAVLA: A Force-Adaptive Fast-Slow VLA model for Contact-Rich Robotic Manipulation](https://mlsanity.com/paper/2602.23648/) (2026)
- [TaF-VLA: Tactile-Force Alignment in Vision-Language-Action Models for Force-aware Manipulation](https://mlsanity.com/paper/2601.20321/) (2026)
- [AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perception](https://mlsanity.com/paper/2602.09617/) (2026)
- [Mind Meets Space: Rethinking Agentic Spatial Intelligence from a Neuroscience-inspired Perspective](https://mlsanity.com/paper/2509.09154/) (2025)
- [ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generation](https://mlsanity.com/paper/2505.20498/) (2025)
- [Object Pose Estimation through Dexterous Touch](https://mlsanity.com/paper/2509.13591/) (2025)
- [The Role of Touch: Towards Optimal Tactile Sensing Distribution in Anthropomorphic Hands for Dexterous In-Hand Manipulation](https://mlsanity.com/paper/2509.14984/) (2025)
- [Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch](https://mlsanity.com/paper/2604.09100/) (2026)
- [Compact Visuotactile World Models for Lifting: Prediction, Reward Alignment, and Force Constraints](https://mlsanity.com/paper/2609.09597/) (2026)
- [What to Attend, What to Keep: Skill-Conditioned Visuotactile Representation with Progress-Guided Event Memory](https://mlsanity.com/paper/2609.38494/) (2026)

## Abstract (arXiv)

In this paper, we tackle the problem of estimating 3D contact forces using vision-based tactile sensors. In particular, our goal is to estimate contact forces over a large range (up to 15 N) on any objects while generalizing across different vision-based tactile sensors. Thus, we collected a dataset of over 200K indentations using a robotic arm that pressed various indenters onto a GelSight Mini sensor mounted on a force sensor and then used the data to train a multi-head transformer for force regression. Strong generalization is achieved via accurate data collection and multi-objective optimization that leverages depth contact images. Despite being trained only on primitive shapes and textures, the regressor achieves a mean absolute error of 4% on a dataset of unseen real-world objects. We further evaluate our approach's generalization capability to other GelSight mini and DIGIT sensors, and propose a reproducible calibration procedure for adapting the pre-trained model to other vision-based sensors. Furthermore, the method was evaluated on real-world tasks, including weighing objects and controlling the deformation of delicate objects, which relies on accurate force feedback. Project webpage: http://prg.cs.umd.edu/FeelAnyForce