# TouchAI: Exploring human-AI perceptual alignment in touch through language model representations

arXiv: https://arxiv.org/abs/2406.06587 · HTML: https://mlsanity.com/paper/2406.06587/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 24 of its 86 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 24 of 24)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Text and Code Embeddings by Contrastive Pre-Training](https://mlsanity.com/paper/2201.10005/) (2022)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark](https://mlsanity.com/paper/2304.03279/) (2023)
- [Human alignment of neural network representations](https://mlsanity.com/paper/2211.01201/) (2022)
- [Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation](https://mlsanity.com/paper/2205.00501/) (2022)
- [A Touch, Vision, and Language Dataset for Multimodal Alignment](https://mlsanity.com/paper/2402.13232/) (2024)
- [Text-To-Concept (and Back) via Cross-Model Alignment](https://mlsanity.com/paper/2305.06386/) (2023)
- [Large language models predict human sensory judgments across six modalities](https://mlsanity.com/paper/2302.01308/) (2023)
- [The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects](https://mlsanity.com/paper/2306.00956/) (2023)
- [Shared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model Behavior](https://mlsanity.com/paper/2107.09234/) (2021)
- [Simulation of Vision-based Tactile Sensors using Physics based Rendering](https://mlsanity.com/paper/2012.13184/) (2020)
- [A Sensorized Multicurved Robot Finger with Data-driven Touch Sensing via Overlapping Light Signals](https://mlsanity.com/paper/2004.00685/) (2020)
- [TextileNet: A Material Taxonomy-based Fashion Textile Dataset](https://mlsanity.com/paper/2301.06160/) (2023)
- [Telextiles: End-to-end Remote Transmission of Fabric Tactile Sensation](https://mlsanity.com/paper/2405.03363/) (2024)

## Built on by (most cited first, 2 of 2)

- [The Zero Body Problem: Probing LLM Use of Sensory Language](https://mlsanity.com/paper/2504.06393/) (2025)
- [Extracting Conceptual Spaces from LLMs Using Prototype Embeddings](https://mlsanity.com/paper/2509.19269/) (2025)

## Abstract (arXiv)

Aligning large language models (LLMs) behaviour with human intent is critical for future AI. An important yet often overlooked aspect of this alignment is the perceptual alignment. Perceptual modalities like touch are more multifaceted and nuanced compared to other sensory modalities such as vision. This work investigates how well LLMs align with human touch experiences using the "textile hand" task. We created a "Guess What Textile" interaction in which participants were given two textile samples -- a target and a reference -- to handle. Without seeing them, participants described the differences between them to the LLM. Using these descriptions, the LLM attempted to identify the target textile by assessing similarity within its high-dimensional embedding space. Our results suggest that a degree of perceptual alignment exists, however varies significantly among different textile samples. For example, LLM predictions are well aligned for silk satin, but not for cotton denim. Moreover, participants didn't perceive their textile experiences closely matched by the LLM predictions. This is only the first exploration into perceptual alignment around touch, exemplified through textile hand. We discuss possible sources of this alignment variance, and how better human-AI perceptual alignment can benefit future everyday tasks.