# Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering

arXiv: https://arxiv.org/abs/1505.05612 · HTML: https://mlsanity.com/paper/1505.05612/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2015.

## Built on by (most cited first, 30 of 80)

- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark](https://mlsanity.com/paper/2410.03051/) (2024)
- [M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning](https://mlsanity.com/paper/2306.04387/) (2023)
- [CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark](https://mlsanity.com/paper/2406.05967/) (2024)
- [Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories](https://mlsanity.com/paper/2306.09224/) (2023)
- [MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering](https://mlsanity.com/paper/2405.11985/) (2024)
- [M6: A Chinese Multimodal Pretrainer](https://mlsanity.com/paper/2103.00823/) (2021)
- [Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering](https://mlsanity.com/paper/2204.09634/) (2022)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [xGQA: Cross-Lingual Visual Question Answering](https://mlsanity.com/paper/2109.06082/) (2021)
- [FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion](https://mlsanity.com/paper/2402.03226/) (2024)
- [VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs](https://mlsanity.com/paper/2101.12059/) (2021)
- [3D Question Answering](https://mlsanity.com/paper/2112.08359/) (2021)
- [ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation](https://mlsanity.com/paper/2112.15283/) (2021)
- [WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines](https://mlsanity.com/paper/2410.12705/) (2024)
- [Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matrices](https://mlsanity.com/paper/2201.12382/) (2022)
- [Multimodal Dialogue Response Generation](https://mlsanity.com/paper/2110.08515/) (2021)
- [Vision-Language Intelligence: Tasks, Representation Learning, and Large Models](https://mlsanity.com/paper/2203.01922/) (2022)
- [SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Models](https://mlsanity.com/paper/2312.09818/) (2023)
- [UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models](https://mlsanity.com/paper/2310.10942/) (2023)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [Robust Visual Question Answering: Datasets, Methods, and Future Challenges](https://mlsanity.com/paper/2307.11471/) (2023)
- [JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation](https://mlsanity.com/paper/2410.17250/) (2024)
- [Multimodal Conversational AI: A Survey of Datasets and Approaches](https://mlsanity.com/paper/2205.06907/) (2022)
- [Asking questions on handwritten document collections](https://mlsanity.com/paper/2110.00711/) (2021)
- [VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challenges](https://mlsanity.com/paper/2212.13296/) (2022)
- [DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue](https://mlsanity.com/paper/2101.00151/) (2021)
- [Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation](https://mlsanity.com/paper/2109.04653/) (2021)
- [GoG: Relation-aware Graph-over-Graph Network for Visual Dialog](https://mlsanity.com/paper/2109.08475/) (2021)

## Abstract (arXiv)

In this paper, we present the mQA model, which is able to answer questions about the content of an image. The answer can be a sentence, a phrase or a single word. Our model contains four components: a Long Short-Term Memory (LSTM) to extract the question representation, a Convolutional Neural Network (CNN) to extract the visual representation, an LSTM for storing the linguistic context in an answer, and a fusing component to combine the information from the first three components and generate the answer. We construct a Freestyle Multilingual Image Question Answering (FM-IQA) dataset to train and evaluate our mQA model. It contains over 150,000 images and 310,000 freestyle Chinese question-answer pairs and their English translations. The quality of the generated answers of our mQA model on this dataset is evaluated by human judges through a Turing Test. Specifically, we mix the answers provided by humans and our model. The human judges need to distinguish our model from the human. They will also provide a score (i.e. 0, 1, 2, the larger the better) indicating the quality of the answer. We propose strategies to monitor the quality of this evaluation process. The experiments show that in 64.7% of cases, the human judges cannot distinguish our model from humans. The average score is 1.454 (1.918 for human). The details of this work, including the FM-IQA dataset, can be found on the project page: http://idl.baidu.com/FM-IQA.html