# Lost in Speech: Benchmarking, Evaluation, and Parsing of Spoken Bilingual Conversational Language Beyond Standard UD Assumptions

arXiv: https://arxiv.org/abs/2602.06307 · HTML: https://mlsanity.com/paper/2602.06307/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 32 references that are in the map.
- First posted on arXiv 6 Feb 2026.

## Builds on (most cited first, 9 of 9)

- [Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing](https://mlsanity.com/paper/2101.03289/) (2021)
- [Universal Dependency Parsing for Hindi-English Code-switching](https://mlsanity.com/paper/1804.05868/) (2018)
- [Treebanking User-Generated Content: a UD Based Overview of Guidelines, Corpora and Unified Recommendations](https://mlsanity.com/paper/2011.02063/) (2020)
- [Minimal Pair-Based Evaluation of Code-Switching](https://mlsanity.com/paper/2506.01840/) (2025)
- [Code-Mixed Probes Show How Pre-Trained Models Generalise On Code-Switched Text](https://mlsanity.com/paper/2403.04872/) (2024)
- [Self-Correction Makes LLMs Better Parsers](https://mlsanity.com/paper/2504.14165/) (2025)
- [Code-Switching and Syntax: A Large-Scale Experiment](https://mlsanity.com/paper/2506.01846/) (2025)
- [A Syntax-Injected Approach for Faster and More Accurate Sentiment Analysis](https://mlsanity.com/paper/2406.15163/) (2024)
- [Dancing in the syntax forest: fast, accurate and explainable sentiment analysis with SALSA](https://mlsanity.com/paper/2406.16071/) (2024)

## Abstract (arXiv)

Spoken bilingual conversations pose substantial challenges for syntactic parsing because they often include disfluencies and discourse-driven structures that complicate dependency parsing under standard Universal Dependencies (UD) assumptions and evaluation practices. To systematically study these challenges, in this work, we first introduce a linguistically grounded taxonomy of conversational bilingual phenomena, together with SpokeBench, an expert-annotated English-Spanish benchmark for structurally complex speech. To address the limitations of existing evaluation practices, we propose Flex-UD, an ambiguity-aware evaluation metric that distinguishes catastrophic structural failures from linguistically acceptable variations. Finally, we introduce DECAP, a decoupled agentic parsing framework that separates spoken-phenomena handling from core syntactic analysis, enabling robust and interpretable dependency parsing without retraining. Experiments across both proprietary and open-weight LLMs show that DECAP substantially improves performance on complex conversational phenomena and achieves over 60% improvements in UPOS-F1 Score over baselines, while Flex-UD evaluations reveal gains that otherwise remain partially hidden under standard attachment-based metrics.