# CI-AVSR: A Cantonese Audio-Visual Speech Dataset for In-car Command Recognition

arXiv: https://arxiv.org/abs/2201.03804 · HTML: https://mlsanity.com/paper/2201.03804/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 37 references that are in the map.
- First posted on arXiv 11 Jan 2022.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021)
- [End-to-end Audiovisual Speech Recognition](https://mlsanity.com/paper/1802.06424/) (2018)
- [Deep Audio-Visual Learning: A Survey](https://mlsanity.com/paper/2001.04758/) (2020)
- [Discriminative Multi-modality Speech Recognition](https://mlsanity.com/paper/2005.05592/) (2020)
- [Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformer](https://mlsanity.com/paper/1910.13923/) (2019)
- [ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversation](https://mlsanity.com/paper/2112.06223/) (2021)
- [Multimodal End-to-End Sparse Model for Emotion Recognition](https://mlsanity.com/paper/2103.09666/) (2021)
- [Survey on Deep Neural Networks in Speech and Vision Systems](https://mlsanity.com/paper/1908.07656/) (2019)
- [Meta-Transfer Learning for Code-Switched Speech Recognition](https://mlsanity.com/paper/2004.14228/) (2020)
- [The Multimodal Sentiment Analysis in Car Reviews (MuSe-CaR) Dataset: Collection, Insights and Improvements](https://mlsanity.com/paper/2101.06053/) (2021)
- [Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition](https://mlsanity.com/paper/2009.09629/) (2020)
- [Factorized Multimodal Transformer for Multimodal Sequential Learning](https://mlsanity.com/paper/1911.09826/) (2019)

## Built on by (most cited first, 10 of 10)

- [IndoRobusta: Towards Robustness Against Diverse Code-Mixed Indonesian Local Languages](https://mlsanity.com/paper/2311.12405/) (2023)
- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)
- [Toward Practical Automatic Speech Recognition and Post-Processing: a Call for Explainable Error Benchmark Guideline](https://mlsanity.com/paper/2401.14625/) (2024)
- [OLKAVS: An Open Large-Scale Korean Audio-Visual Speech Dataset](https://mlsanity.com/paper/2301.06375/) (2023)
- [Cross-Lingual Cross-Age Group Adaptation for Low-Resource Elderly Speech Emotion Recognition](https://mlsanity.com/paper/2306.14517/) (2023)
- [Kaggle Competition: Cantonese Audio-Visual Speech Recognition for In-car Commands](https://mlsanity.com/paper/2207.02663/) (2022)
- [Model-free Speculative Decoding for Transformer-based ASR with Token Map Drafting](https://mlsanity.com/paper/2507.21522/) (2025)
- [ReasonBreak: Probing Vulnerabilities in Reasoning-Enabled Vision-Language-Action Models for Autonomous Driving](https://mlsanity.com/paper/2605.29114/) (2026)
- [Candor-LR: A Dyadic Conversational Dataset for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2609.10394/) (2026)
- [Improving Audiovisual Speech Recognition through Synthetic Visual Data Augmentation](https://mlsanity.com/paper/2609.31961/) (2026)

## Abstract (arXiv)

With the rise of deep learning and intelligent vehicle, the smart assistant has become an essential in-car component to facilitate driving and provide extra functionalities. In-car smart assistants should be able to process general as well as car-related commands and perform corresponding actions, which eases driving and improves safety. However, there is a data scarcity issue for low resource languages, hindering the development of research and applications. In this paper, we introduce a new dataset, Cantonese In-car Audio-Visual Speech Recognition (CI-AVSR), for in-car command recognition in the Cantonese language with both video and audio data. It consists of 4,984 samples (8.3 hours) of 200 in-car commands recorded by 30 native Cantonese speakers. Furthermore, we augment our dataset using common in-car background noises to simulate real environments, producing a dataset 10 times larger than the collected one. We provide detailed statistics of both the clean and the augmented versions of our dataset. Moreover, we implement two multimodal baselines to demonstrate the validity of CI-AVSR. Experiment results show that leveraging the visual signal improves the overall performance of the model. Although our best model can achieve a considerable quality on the clean test set, the speech recognition quality on the noisy data is still inferior and remains as an extremely challenging task for real in-car speech recognition systems. The dataset and code will be released at https://github.com/HLTCHKUST/CI-AVSR.