# HeiSD: Hybrid Speculative Decoding for Embodied Vision-Language-Action Models with Kinematic Awareness

arXiv: https://arxiv.org/abs/2603.17573 · HTML: https://mlsanity.com/paper/2603.17573/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Efficient VLA inference and adaptation, one of 13 directions in Vision-language-action models.
- Builds on 33 of its 46 references that are in the map.
- First posted on arXiv 18 Mar 2026.

## Placed in

- Vision-language-action models / [Efficient VLA inference and adaptation](https://mlsanity.com/vla/efficient-vla-inference-and-adaptation/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 30 of 33)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)
- [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://mlsanity.com/paper/2406.02523/) (2024)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2409.12514/) (2024)
- [A Survey on Vision-Language-Action Models for Embodied AI](https://mlsanity.com/paper/2405.14093/) (2024)
- [REST: Retrieval-Based Speculative Decoding](https://mlsanity.com/paper/2311.08252/) (2023)
- [DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution](https://mlsanity.com/paper/2411.02359/) (2024)
- [MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulation](https://mlsanity.com/paper/2503.20384/) (2025)
- [MBQ: Modality-Balanced Quantization for Large Vision-Language Models](https://mlsanity.com/paper/2412.19509/) (2024)
- [CEED-VLA: Consistency Vision-Language-Action Model with Early-Exit Decoding](https://mlsanity.com/paper/2506.13725/) (2025)
- [Decoding Speculative Decoding](https://mlsanity.com/paper/2402.01528/) (2024)
- [Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptance](https://mlsanity.com/paper/2507.22424/) (2025)
- [QVLA: Not All Channels Are Equal in Vision-Language-Action Model's Quantization](https://mlsanity.com/paper/2602.03782/) (2026)
- [Quantization-Aware Imitation-Learning for Resource-Efficient Robotic Control](https://mlsanity.com/paper/2412.01034/) (2024)
- [EdgeVLA: Efficient Vision-Language-Action Models](https://mlsanity.com/paper/2507.14049/) (2025)
- [Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration](https://mlsanity.com/paper/2412.00061/) (2024)
- [DyQ-VLA: Temporal-Dynamic-Aware Quantization for Embodied Vision-Language-Action Models](https://mlsanity.com/paper/2603.07904/) (2026)
- [Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match](https://mlsanity.com/paper/2511.22972/) (2025)
- [KERV: Kinematic-Rectified Speculative Decoding for Embodied VLA Models](https://mlsanity.com/paper/2603.01581/) (2026)
- [SpecPV: Improving Self-Speculative Decoding for Long-Context Generation via Partial Verification](https://mlsanity.com/paper/2512.02337/) (2025)

## Built on by (most cited first, 6 of 6)

- [RoboECC: Multi-Factor-Aware Edge-Cloud Collaborative Deployment for VLA Models](https://mlsanity.com/paper/2603.20711/) (2026)
- [RAP: Retrieve, Adapt, and Prompt-Fit for Training-Free Few-Shot Medical Image Segmentation](https://mlsanity.com/paper/2603.27705/) (2026)
- [2D or 3D: Who Governs Salience in VLA Models? -- Tri-Stage Token Pruning Framework with Modality Salience Awareness](https://mlsanity.com/paper/2604.09244/) (2026)
- [Deltoris: Enabling Real-time VLA Inference in Embodied AI via Bit-level Sparsity and Speculative Inference](https://mlsanity.com/paper/2608.04428/) (2026)
- [WA-SpecDec: World-Aware Speculative Decoding for Vision-Language-Action Models](https://mlsanity.com/paper/2608.08725/) (2026)
- [Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosis](https://mlsanity.com/paper/2608.20743/) (2026)

## Abstract (arXiv)

Vision-Language-Action (VLA) Models have become the mainstream solution for robot control, but suffer from slow inference speeds. Speculative Decoding (SD) is a promising acceleration method which can be divided into two categories: drafter-based SD and retrieval-based SD. Each of the two methods demonstrates complementary advantages and limitations when applied to VLA models, leading to the hypothesis that a hybrid approach integrating these two methods will yield better performance. In this paper, we first conduct a series of detailed analyses to reveal the advantages and feasibility of hybrid utilization. However, even with the aforementioned key insights, implementing hybrid SD in VLA models presents several challenges: (1) draft rejection and persistent errors in retrieval-based SD; (2) difficulty in determining the hybrid boundary. To address these, we propose the HeiSD framework. We propose a retrieval-based SD optimization method in HeiSD, which contains a verify-skip mechanism and a sequence-wise relaxed acceptance strategy. Moreover, we proposed a kinematic-based fused metric in HeiSD to automatically determine the hybrid boundary. Experimental results demonstrate that HeiSD attains a speedup of up to 2.45x in simulation benchmarks and 2.06x~2.41x in real-world scenarios, while sustaining a high task success rate.