# Multimodal Structure-Aware Quantum Data Processing

arXiv: https://arxiv.org/abs/2411.04242 · HTML: https://mlsanity.com/paper/2411.04242/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 22 references that are in the map.
- First posted on arXiv 6 Nov 2024.

## Builds on (most cited first, 7 of 7)

- [Probing Image-Language Transformers for Verb Understanding](https://mlsanity.com/paper/2106.09141/) (2021)
- [QNLP in Practice: Running Compositional Models of Meaning on a Quantum Computer](https://mlsanity.com/paper/2102.12846/) (2021)
- [lambeq: An Efficient High-Level Python Library for Quantum NLP](https://mlsanity.com/paper/2110.04236/) (2021)
- [Quantum Natural Language Processing on Near-Term Quantum Computers](https://mlsanity.com/paper/2005.04147/) (2020)
- [Grammar-aware sentence classification on quantum computers](https://mlsanity.com/paper/2012.03756/) (2020)
- [Towards Transparency in Coreference Resolution: A Quantum-Inspired Approach](https://mlsanity.com/paper/2312.00688/) (2023)
- [Multimodal Quantum Natural Language Processing: A Novel Framework for using Quantum Methods to Analyse Real Data](https://mlsanity.com/paper/2411.05023/) (2024)

## Abstract (arXiv)

While large language models (LLMs) have advanced the field of natural language processing (NLP), their "black box" nature obscures their decision-making processes. To address this, researchers developed structured approaches using higher order tensors. These are able to model linguistic relations, but stall when training on classical computers due to their excessive size. Tensors are natural inhabitants of quantum systems and training on quantum computers provides a solution by translating text to variational quantum circuits. In this paper, we develop MultiQ-NLP: a framework for structure-aware data processing with multimodal text+image data. Here, "structure" refers to syntactic and grammatical relationships in language, as well as the hierarchical organization of visual elements in images. We enrich the translation with new types and type homomorphisms and develop novel architectures to represent structure. When tested on a main stream image classification task (SVO Probes), our best model showed a par performance with the state of the art classical models; moreover the best model was fully structured.