# WASIL: In-the-Wild Arabic Spoken Interactions with LLMs

arXiv: https://arxiv.org/abs/2605.16364 · HTML: https://mlsanity.com/paper/2605.16364/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 64 references that are in the map.
- First posted on arXiv 9 May 2026.

## Builds on (most cited first, 30 of 38)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [OR-Bench: An Over-Refusal Benchmark for Large Language Models](https://mlsanity.com/paper/2405.20947/) (2024)
- [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017)
- [VoiceBench: Benchmarking LLM-Based Voice Assistants](https://mlsanity.com/paper/2410.17196/) (2024)
- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://mlsanity.com/paper/2305.14456/) (2023)
- [Cultural Bias and Cultural Alignment of Large Language Models](https://mlsanity.com/paper/2311.14096/) (2023)
- [Fanar: An Arabic-Centric Multimodal Generative AI Platform](https://mlsanity.com/paper/2501.13944/) (2025)
- [ALLaM: Large Language Models for Arabic and English](https://mlsanity.com/paper/2407.15390/) (2024)
- [Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding](https://mlsanity.com/paper/2104.02138/) (2021)
- [Out-of-Scope Intent Detection with Self-Supervision and Discriminative Training](https://mlsanity.com/paper/2106.08616/) (2021)
- [Towards One Model to Rule All: Multilingual Strategy for Dialectal Code-Switching Arabic ASR](https://mlsanity.com/paper/2105.14779/) (2021)
- [Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning](https://mlsanity.com/paper/2510.12712/) (2025)
- [A Survey on Asking Clarification Questions Datasets in Conversational Systems](https://mlsanity.com/paper/2305.15933/) (2023)
- [PalmX 2025: The First Shared Task on Benchmarking LLMs on Arabic and Islamic Culture](https://mlsanity.com/paper/2509.02550/) (2025)
- [Casablanca: Data and Models for Multidialectal Arabic Speech Recognition](https://mlsanity.com/paper/2410.04527/) (2024)
- [ZAEBUC-Spoken: A Multilingual Multidialectal Arabic-English Speech Corpus](https://mlsanity.com/paper/2403.18182/) (2024)
- [SpokenNativQA: Multilingual Everyday Spoken Queries for LLMs](https://mlsanity.com/paper/2505.19163/) (2025)
- [SOVA-Bench: Benchmarking the Speech Conversation Ability for LLM-based Voice Assistant](https://mlsanity.com/paper/2506.02457/) (2025)
- [VoiceAssistant-Eval: Benchmarking AI Assistants across Listening, Speaking, and Viewing](https://mlsanity.com/paper/2509.22651/) (2025)

## Built on by (most cited first, 1 of 1)

- [ImageEval 2026: Culturally Grounded Arabic Multimodal Evaluation](https://mlsanity.com/paper/2608.30475/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) voice assistants are commonly built as cascaded Automatic Speech recognition (ASR) to LLM systems, where recognition errors can distort user intent. Dislikes may also arise from ambiguous, out-of-domain, or non-request turns, making it hard to isolate ASR effects. We release WASIL (it denotes connection or linking in Arabic): in-the-wild Arabic spoken interaction prompts with audio, ASR hypotheses, assistant responses, and explicit like/dislike feedback (8,529 turns; 14.2% dislikes), plus a 2,000-turn test set covering Modern Standard Arabic (MSA) and four major dialects with their labels. We provide low-cost gold transcripts via multi-ASR agreement-guided post-editing and annotate answerability (answerable, ambiguous/needs-clarification, unsupported, not-a-request/noise) to separate intrinsic unanswerability from ASR-induced degradation. Finally, we describe scalable reference-free evaluation of responses from ASR vs. gold transcripts using multi-judge LLM scoring.