mlsanityData to 14 Oct 2026

Talk to Me, Jarvis: An Open-Source Edge-Deployable Voice Assistant Framework for Autonomous Racecars

arXiv 2609.21109 ↗First posted 17 Sep 2026cs.LG cs.ROcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent advances in large language models have improved their effectiveness as back-end components for voice assistants, particularly in intent understanding and context-aware input classification. However, online-hosted models introduce network dependency and variable inference latency, limiting their suitability for time-critical autonomous driving applications. In this work, we address these issues by developing Jarvis, an offline voice assistant for high-level behavioral commands of autonomous vehicles. Its architecture integrates speech recognition and synthesis with natural language command classification into a lightweight, local framework. Jarvis core component is a text-to-command classifier, built using a domain-specific fine-tuning of the Mistral 7B model, demonstrating low-latency inference. Our experimental evaluation demonstrates that our solution outperforms larger online-hosted models, achieving 97.63 % intent recognition accuracy with an average processing latency of 1.39 s, making it well-suited for operations requiring quick response times. To support further research and fine-tuning, we provide an open-source implementation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 38 references that are in the map. First posted on arXiv 17 Sep 2026.

Builds on

32 of 38 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  9. 2020GLU Variants Improve Transformercited by 1,804
  10. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  11. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  12. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
20 more
  1. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  2. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  3. 2023GPT-Driver: Learning to Drive with GPTcited by 347
  4. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  5. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 232
  6. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 190
  7. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 139
  8. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  9. 2021AEDA: An Easier Data Augmentation Technique for Text Classificationcited by 67
  10. 2022Autonomous Vehicles on the Edge: A Survey on Autonomous Vehicle Racingcited by 64
  11. 2024Driving Everywhere with Large Language Model Policy Adaptationcited by 53
  12. 2019Soft Contextual Data Augmentation for Neural Machine Translationcited by 49
  13. 2024A Comprehensive Survey on Data Augmentationcited by 40
  14. 2023Human-Centric Autonomous Systems With LLMs for User Command Reasoningcited by 22
  15. 2024From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systemscited by 17
  16. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trendscited by 10
  17. 2025Automatic Speech Recognition in the Modern Era: Architectures, Training, and Evaluationcited by 3
  18. 2025Improving Speech Recognition Accuracy Using Custom Language Models with the Vosk Toolkitcited by 1
  19. 2026Modular Autonomy with Conversational Interaction: An LLM-driven Framework for Decision Making in Autonomous Drivingcited by 1
  20. 2026Head-to-Head autonomous racing at the limits of handling in the A2RL challengecited by 1

Built on by

Not cited yet.