# BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detection

arXiv: https://arxiv.org/abs/1902.00038 · HTML: https://mlsanity.com/paper/1902.00038/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2019.

## Built on by (most cited first, 30 of 62)

- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering](https://mlsanity.com/paper/2104.03149/) (2021)
- [Invariant Grounding for Video Question Answering](https://mlsanity.com/paper/2206.02349/) (2022)
- [DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation](https://mlsanity.com/paper/2101.05544/) (2021)
- [SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgery](https://mlsanity.com/paper/2304.09974/) (2023)
- [Structured Sparse R-CNN for Direct Scene Graph Generation](https://mlsanity.com/paper/2106.10815/) (2021)
- [Spatial Dual-Modality Graph Reasoning for Key Information Extraction](https://mlsanity.com/paper/2103.14470/) (2021)
- [Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sports](https://mlsanity.com/paper/2401.01505/) (2024)
- [Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery](https://mlsanity.com/paper/2405.10948/) (2024)
- [EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery](https://mlsanity.com/paper/2501.11347/) (2025)
- [MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2203.09138/) (2022)
- [Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answering](https://mlsanity.com/paper/2103.05568/) (2021)
- [WizMap: Scalable Interactive Visualization for Exploring Large Machine Learning Embeddings](https://mlsanity.com/paper/2306.09328/) (2023)
- [RTIC: Residual Learning for Text and Image Composition using Graph Convolutional Network](https://mlsanity.com/paper/2104.03015/) (2021)
- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Tasks](https://mlsanity.com/paper/2204.07780/) (2022)
- [Locate before Answering: Answer Guided Question Localization for Video Question Answering](https://mlsanity.com/paper/2210.02081/) (2022)
- [Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problem](https://mlsanity.com/paper/2207.11850/) (2022)
- [LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answering](https://mlsanity.com/paper/2105.14300/) (2021)
- [CAT-ViL: Co-Attention Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery](https://mlsanity.com/paper/2307.05182/) (2023)
- [Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgery](https://mlsanity.com/paper/2408.04958/) (2024)
- [OmniV-Med: Scaling Medical Vision-Language Model for Universal Visual Understanding](https://mlsanity.com/paper/2504.14692/) (2025)
- [Image Scene Graph Generation (SGG) Benchmark](https://mlsanity.com/paper/2107.12604/) (2021)
- [VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challenges](https://mlsanity.com/paper/2212.13296/) (2022)
- [A survey on VQA_Datasets and Approaches](https://mlsanity.com/paper/2105.00421/) (2021)
- [Leveraging Predicate and Triplet Learning for Scene Graph Generation](https://mlsanity.com/paper/2406.02038/) (2024)
- [A Comprehensive Survey on Visual Question Answering Datasets and Algorithms](https://mlsanity.com/paper/2411.11150/) (2024)
- [Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answering](https://mlsanity.com/paper/2209.06954/) (2022)

## Abstract (arXiv)

Multimodal representation learning is gaining more and more interest within the deep learning community. While bilinear models provide an interesting framework to find subtle combination of modalities, their number of parameters grows quadratically with the input dimensions, making their practical implementation within classical deep learning pipelines challenging. In this paper, we introduce BLOCK, a new multimodal fusion based on the block-superdiagonal tensor decomposition. It leverages the notion of block-term ranks, which generalizes both concepts of rank and mode ranks for tensors, already used for multimodal fusion. It allows to define new ways for optimizing the tradeoff between the expressiveness and complexity of the fusion model, and is able to represent very fine interactions between modalities while maintaining powerful mono-modal representations. We demonstrate the practical interest of our fusion model by using BLOCK for two challenging tasks: Visual Question Answering (VQA) and Visual Relationship Detection (VRD), where we design end-to-end learnable architectures for representing relevant interactions between modalities. Through extensive experiments, we show that BLOCK compares favorably with respect to state-of-the-art multimodal fusion models for both VQA and VRD tasks. Our code is available at https://github.com/Cadene/block.bootstrap.pytorch.