# MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering

arXiv: https://arxiv.org/abs/2010.14095 · HTML: https://mlsanity.com/paper/2010.14095/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2020.

## Built on by (most cited first, 30 of 30)

- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)
- [Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey](https://mlsanity.com/paper/2302.10035/) (2023)
- [Video Question Answering: Datasets, Algorithms and Challenges](https://mlsanity.com/paper/2203.01225/) (2022)
- [Kaleido-BERT: Vision-Language Pre-training on Fashion Domain](https://mlsanity.com/paper/2103.16110/) (2021)
- [TubeR: Tubelet Transformer for Video Action Detection](https://mlsanity.com/paper/2104.00969/) (2021)
- [Local-Global Context Aware Transformer for Language-Guided Video Segmentation](https://mlsanity.com/paper/2203.09773/) (2022)
- [Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models](https://mlsanity.com/paper/2408.12763/) (2024)
- [Found a Reason for me? Weakly-supervised Grounded Visual Question Answering using Capsules](https://mlsanity.com/paper/2105.04836/) (2021)
- [End-to-End Video Question Answering with Frame Scoring Mechanisms and Adaptive Sampling](https://mlsanity.com/paper/2407.15047/) (2024)
- [VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering](https://mlsanity.com/paper/2109.13116/) (2021)
- [Learning Situation Hyper-Graphs for Video Question Answering](https://mlsanity.com/paper/2304.08682/) (2023)
- [ANetQA: A Large-scale Benchmark for Fine-grained Compositional Reasoning over Untrimmed Videos](https://mlsanity.com/paper/2305.02519/) (2023)
- [HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Language](https://mlsanity.com/paper/2305.17690/) (2023)
- [Knowledge Mining with Scene Text for Fine-Grained Recognition](https://mlsanity.com/paper/2203.14215/) (2022)
- [Semantic Structure based Query Graph Prediction for Question Answering over Knowledge Graph](https://mlsanity.com/paper/2204.10194/) (2022)
- [On the hidden treasure of dialog in video question answering](https://mlsanity.com/paper/2103.14517/) (2021)
- [Neural-Symbolic VideoQA: Learning Compositional Spatio-Temporal Reasoning for Real-world Video Question Answering](https://mlsanity.com/paper/2404.04007/) (2024)
- [ReasVQA: Advancing VideoQA with Imperfect Reasoning Process](https://mlsanity.com/paper/2501.13536/) (2025)
- [DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention](https://mlsanity.com/paper/2210.16431/) (2022)
- [ParCNetV2: Oversized Kernel with Enhanced Attention](https://mlsanity.com/paper/2211.07157/) (2022)
- [M-SENSE: Modeling Narrative Structure in Short Personal Narratives Using Protagonist's Mental Representations](https://mlsanity.com/paper/2302.09418/) (2023)
- [Can Model Fusing Help Transformers in Long Document Classification? An Empirical Study](https://mlsanity.com/paper/2307.09532/) (2023)
- [The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering](https://mlsanity.com/paper/2501.07109/) (2025)
- [FAR-Net: Multi-Stage Fusion Network with Enhanced Semantic Alignment and Adaptive Reconciliation for Composed Image Retrieval](https://mlsanity.com/paper/2507.12823/) (2025)
- [Learning Fine-Grained Visual Understanding for Video Question Answering via Decoupling Spatial-Temporal Modeling](https://mlsanity.com/paper/2210.03941/) (2022)
- [SceneGATE: Scene-Graph based co-Attention networks for TExt visual question answering](https://mlsanity.com/paper/2212.08283/) (2022)
- [Improved RAMEN: Towards Domain Generalization for Visual Question Answering](https://mlsanity.com/paper/2109.02370/) (2021)
- [Self-Regulated Data-Free Knowledge Amalgamation for Text Classification](https://mlsanity.com/paper/2406.15476/) (2024)
- [Enhancing Weakly Supervised Multimodal Video Anomaly Detection through Text Guidance](https://mlsanity.com/paper/2602.10549/) (2026)
- [Leveraging Static Relationships for Intra-Type and Inter-Type Message Passing in Video Question Answering](https://mlsanity.com/paper/2504.02417/) (2025)

## Abstract (arXiv)

We present MMFT-BERT(MultiModal Fusion Transformer with BERT encodings), to solve Visual Question Answering (VQA) ensuring individual and combined processing of multiple input modalities. Our approach benefits from processing multimodal data (video and text) adopting the BERT encodings individually and using a novel transformer-based fusion method to fuse them together. Our method decomposes the different sources of modalities, into different BERT instances with similar architectures, but variable weights. This achieves SOTA results on the TVQA dataset. Additionally, we provide TVQA-Visual, an isolated diagnostic subset of TVQA, which strictly requires the knowledge of visual (V) modality based on a human annotator's judgment. This set of questions helps us to study the model's behavior and the challenges TVQA poses to prevent the achievement of super human performance. Extensive experiments show the effectiveness and superiority of our method.