# SignVLA: Real-Time Sign Language-Guided Robotic Manipulation via Attention LSTM and Vision-Language-Action Models

arXiv: https://arxiv.org/abs/2606.20857 · HTML: https://mlsanity.com/paper/2606.20857/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Generalist VLA models and benchmarks, one of 13 directions in Vision-language-action models.
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 18 Jun 2026.

## Placed in

- Vision-language-action models / [Generalist VLA models and benchmarks](https://mlsanity.com/vla/generalist-vla-models-and-benchmarks/)

## Builds on (most cited first, 15 of 15)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [MediaPipe Hands: On-device Real-time Hand Tracking](https://mlsanity.com/paper/2006.10214/) (2020)
- [Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining](https://mlsanity.com/paper/2307.14768/) (2023)
- [ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognition](https://mlsanity.com/paper/2304.05934/) (2023)
- [An Efficient Sign Language Translation Using Spatial Configuration and Motion Dynamics with LLMs](https://mlsanity.com/paper/2408.10593/) (2024)
- [Real-time object detection and robotic manipulation for agriculture using a YOLO-based learning approach](https://mlsanity.com/paper/2401.15785/) (2024)
- [Discrete-Time Stress Matrix-Based Formation Control of General Linear Multi-Agent Systems](https://mlsanity.com/paper/2401.05083/) (2024)
- [Signformer is all you need: Towards Edge AI for Sign Language](https://mlsanity.com/paper/2411.12901/) (2024)
- [Deep Reinforcement Learning Optimization for Uncertain Nonlinear Systems via Event-Triggered Robust Adaptive Dynamic Programming](https://mlsanity.com/paper/2512.15735/) (2025)
- [Disturbance observer-based tracking control for roll-to-roll slot die coating systems under gap and pump rate disturbances](https://mlsanity.com/paper/2601.08488/) (2026)

## Abstract (arXiv)

Vision-Language-Action (VLA) models enable robots to execute manipulation tasks from natural-language instructions grounded in visual observations. However, existing VLA interfaces primarily rely on speech or text input, limiting accessibility for deaf, hard-of-hearing, and speech-impaired users. We present SignVLA, a real-time sign-language-guided VLA framework for accessible human-robot interaction. The system introduces a modular sign-to-text interface that converts visual sign gestures into semantic instructions compatible with downstream VLA policies. Given video streams, SignVLA extracts hand landmark features and employs an attention-enhanced Long Short-Term Memory (LSTM) network to capture temporal gesture dynamics for alphabet- and command-level sign recognition. A temporal stabilization module further improves prediction consistency in real-time interaction settings.The generated instruction sequence is then passed to a downstream VLA policy for sign-conditioned robotic manipulation. Experimental results demonstrate stable real-time sign recognition and successful execution of manipulation tasks driven by sign-language inputs. Our findings suggest that lightweight temporal sign recognition can serve as an effective and practical accessibility layer for multimodal embodied intelligence.