# CCFQA: A Benchmark for Cross-Lingual and Cross-Modal Speech and Text Factuality Evaluation

arXiv: https://arxiv.org/abs/2508.07295 · HTML: https://mlsanity.com/paper/2508.07295/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 10 Aug 2025.

## Builds on (most cited first, 14 of 14)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [A Survey of Hallucination in Large Foundation Models](https://mlsanity.com/paper/2309.05922/) (2023)
- [VoiceBench: Benchmarking LLM-Based Voice Assistants](https://mlsanity.com/paper/2410.17196/) (2024)
- [SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training](https://mlsanity.com/paper/2412.15649/) (2024)
- [CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models](https://mlsanity.com/paper/2310.08753/) (2023)
- [SD-QA: Spoken Dialectal Question Answering for the Real World](https://mlsanity.com/paper/2109.12072/) (2021)
- [Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Study](https://mlsanity.com/paper/2502.02481/) (2025)
- [ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generation](https://mlsanity.com/paper/2503.07010/) (2025)
- [Making LLMs Better Many-to-Many Speech-to-Text Translators with Curriculum Learning](https://mlsanity.com/paper/2409.19510/) (2024)
- [Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering](https://mlsanity.com/paper/2505.16591/) (2025)

## Built on by (most cited first, 5 of 5)

- [Evaluating Hallucinations in Audio-Visual Multimodal LLMs with Spoken Queries under Diverse Acoustic Conditions](https://mlsanity.com/paper/2510.08581/) (2025)
- [Safer in Translation? Presupposition Robustness in Indic Languages](https://mlsanity.com/paper/2511.01360/) (2025)
- [MCGA: A Multi-task Classical Chinese Literary Genre Audio Corpus](https://mlsanity.com/paper/2601.09270/) (2026)
- [Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion](https://mlsanity.com/paper/2602.21646/) (2026)
- [A Comprehensive Study of Implementation Bugs in Multi-modal Agents](https://mlsanity.com/paper/2607.04974/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) are increasingly popularized in the multilingual world, ensuring hallucination-free factuality becomes markedly crucial. However, existing benchmarks for evaluating the reliability of Multimodal Large Language Models (MLLMs) predominantly focus on textual or visual modalities with a primary emphasis on English, which creates a gap in evaluation when processing multilingual input, especially in speech. To bridge this gap, we propose a novel Cross-lingual and Cross-modal Factuality benchmark (CCFQA). Specifically, the CCFQA benchmark contains parallel speech-text factual questions across 8 languages, designed to systematically evaluate MLLMs' cross-lingual and cross-modal factuality capabilities. Our experimental results demonstrate that current MLLMs still face substantial challenges on the CCFQA benchmark. Furthermore, we propose a few-shot transfer learning strategy that effectively transfers the Question Answering (QA) capabilities of LLMs in English to multilingual Spoken Question Answering (SQA) tasks, achieving competitive performance with GPT-4o-mini-Audio using just 5-shot training. We release CCFQA as a foundational research resource to promote the development of MLLMs with more robust and reliable speech understanding capabilities. Our code and dataset are available at https://github.com/yxduir/ccfqa.