# AS2D: Accelerating On-Demand Audio Understanding on Mobile Devices

arXiv: https://arxiv.org/abs/2609.37617 · HTML: https://mlsanity.com/paper/2609.37617/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 34 of its 52 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 30 of 34)

- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [Qwen3-ASR Technical Report](https://mlsanity.com/paper/2601.21337/) (2026)
- [Online Speculative Decoding](https://mlsanity.com/paper/2310.07177/) (2023)
- [Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding](https://mlsanity.com/paper/2402.12374/) (2024)
- [PEARL: Parallel Speculative Decoding with Adaptive Draft Length](https://mlsanity.com/paper/2408.11850/) (2024)
- [Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension](https://mlsanity.com/paper/1804.00320/) (2018)
- [MNN: A Universal and Efficient Inference Engine](https://mlsanity.com/paper/2002.12418/) (2020)
- [Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference](https://mlsanity.com/paper/2506.09501/) (2025)
- [Memoro: Using Large Language Models to Realize a Concise Interface for Real-Time Memory Augmentation](https://mlsanity.com/paper/2403.02135/) (2024)
- [Speculative Speculative Decoding](https://mlsanity.com/paper/2603.03251/) (2026)
- [Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference](https://mlsanity.com/paper/2405.14105/) (2024)
- [SGDRC: Software-Defined Dynamic Resource Control for Concurrent DNN Inference on NVIDIA GPUs](https://mlsanity.com/paper/2407.13996/) (2024)
- [Whisper in Medusa's Ear: Multi-head Efficient Decoding for Transformer-based ASR](https://mlsanity.com/paper/2409.15869/) (2024)
- [Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning](https://mlsanity.com/paper/2603.01639/) (2026)
- [WhisperKit: On-device Real-time ASR with Billion-Scale Transformers](https://mlsanity.com/paper/2507.10860/) (2025)
- [SpecASR: Accelerating LLM-based Automatic Speech Recognition via Speculative Decoding](https://mlsanity.com/paper/2507.18181/) (2025)
- [Multi-Drafter Speculative Decoding with Alignment Feedback](https://mlsanity.com/paper/2604.05417/) (2026)
- [OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs](https://mlsanity.com/paper/2606.07577/) (2026)
- [A Scene-aware Models Adaptation Scheme for Cross-scene Online Inference on Mobile Devices](https://mlsanity.com/paper/2407.03331/) (2024)
- [Speculative Speech Recognition by Audio-Prefixed Low-Rank Adaptation of Language Models](https://mlsanity.com/paper/2407.04641/) (2024)
- [Segmentwise Pruning in Audio-Language Models](https://mlsanity.com/paper/2511.14293/) (2025)
- [Locality Matters for Training-Free Audio Token Compression in Audio-Language Models](https://mlsanity.com/paper/2605.25179/) (2026)
- [Event-Grounded Question Answering over Long Audio via Structured Retrieval](https://mlsanity.com/paper/2602.14612/) (2026)

## Abstract (arXiv)

Speculative decoding accelerates autoregressive generation by using a smaller drafter to propose tokens for batched verification by a larger target. However, conventional speculative decoding couples drafting to the target's evolving verified prefix, serializing drafting and verification. We ask whether this dependency is necessary for source-conditioned generation. Our key observation is that, for audio language models, the input audio and user request can provide useful speculative candidates without following the target's evolving text prefix. We propose AS2D (Audio Speculative Speculative Decoding), which enables target-decoupled drafting: an audio-conditioned drafter follows its own generation history while the target independently verifies and corrects ready candidates. Without usable candidates, the target advances alone. Thus, target feedback determines which candidates are committed but no longer determines when the drafter can make progress, enabling drafting and verification to proceed concurrently while retaining target-side verification and correction. We implement AS2D in MNN for Android and evaluate two target models across four phones, seven datasets, and three tasks covering 12.2 hours of audio. Across four phones, AS2D improves pooled ASR throughput by 42-76% over target-only decoding, while only 5.7% of evaluation windows are slower than target-only, compared with 58.1-63.0% for speculative baselines. For ASR, AS2D reaches 97.33-98.20% of a hindsight per-window oracle's pooled throughput over the evaluated drafter/budget catalog. Native on-demand execution with a 7B target achieves up to 78% higher throughput than target-only. These results show that source-conditioned audio generation can relax the conventional dependence of speculative drafting on the target's evolving output prefix, exposing substantial parallelism for efficient inference.