# Hierarchical Cross-Modality Semantic Correlation Learning Model for Multimodal Summarization

arXiv: https://arxiv.org/abs/2112.12072 · HTML: https://mlsanity.com/paper/2112.12072/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [Extractive Summarization of Long Documents by Combining Global and Local Context](https://mlsanity.com/paper/1909.08089/) (2019)
- [VMSMO: Learning to Generate Multimodal Summary for Video-based News Articles](https://mlsanity.com/paper/2010.05406/) (2020)
- [Multimodal Abstractive Summarization for How2 Videos](https://mlsanity.com/paper/1906.07901/) (2019)
- [NEZHA: Neural Contextualized Representation for Chinese Language Understanding](https://mlsanity.com/paper/1909.00204/) (2019)
- [Neural Latent Extractive Document Summarization](https://mlsanity.com/paper/1808.07187/) (2018)
- [Abstractive Text Summarization by Incorporating Reader Comments](https://mlsanity.com/paper/1812.05407/) (2018)
- [Multi-modal Summarization for Video-containing Documents](https://mlsanity.com/paper/2009.08018/) (2020)
- [MAST: Multimodal Abstractive Summarization with Trimodal Hierarchical Attention](https://mlsanity.com/paper/2010.08021/) (2020)

## Built on by (most cited first, 18 of 18)

- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization](https://mlsanity.com/paper/2212.07672/) (2022)
- [MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos](https://mlsanity.com/paper/2306.04216/) (2023)
- [Exploiting Pseudo Image Captions for Multimodal Summarization](https://mlsanity.com/paper/2305.05496/) (2023)
- [Fine-grained and Explainable Factuality Evaluation for Multimodal Summarization](https://mlsanity.com/paper/2402.11414/) (2024)
- [Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment](https://mlsanity.com/paper/2210.04722/) (2022)
- [FineFake: A Knowledge-Enriched Dataset for Fine-Grained Multi-Domain Fake News Detection](https://mlsanity.com/paper/2404.01336/) (2024)
- [PostDoc: Generating Poster from a Long Multimodal Document Using Deep Submodular Optimization](https://mlsanity.com/paper/2405.20213/) (2024)
- [Collaborative Evolution: Multi-Round Learning Between Large and Small Language Models for Emergent Fake News Detection](https://mlsanity.com/paper/2503.21127/) (2025)
- [CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning](https://mlsanity.com/paper/2602.19605/) (2026)
- [Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization](https://mlsanity.com/paper/2408.03149/) (2024)
- [TLDW: Extreme Multimodal Summarisation of News Videos](https://mlsanity.com/paper/2210.08481/) (2022)
- [CISum: Learning Cross-modality Interaction to Enhance Multimodal Semantic Coverage for Multimodal Summarization](https://mlsanity.com/paper/2302.09934/) (2023)
- [D2TV: Dual Knowledge Distillation and Target-oriented Vision Modeling for Many-to-Many Multimodal Summarization](https://mlsanity.com/paper/2305.12767/) (2023)
- [Modeling Paragraph-Level Vision-Language Semantic Alignment for Multi-Modal Summarization](https://mlsanity.com/paper/2208.11303/) (2022)
- [Grafting Pre-trained Models for Multimodal Headline Generation](https://mlsanity.com/paper/2211.07210/) (2022)
- [Labeling Comic Mischief Content in Online Videos with a Multimodal Hierarchical-Cross-Attention Model](https://mlsanity.com/paper/2406.07841/) (2024)
- [SITransformer: Shared Information-Guided Transformer for Extreme Multimodal Summarization](https://mlsanity.com/paper/2408.15829/) (2024)

## Abstract (arXiv)

Multimodal summarization with multimodal output (MSMO) generates a summary with both textual and visual content. Multimodal news report contains heterogeneous contents, which makes MSMO nontrivial. Moreover, it is observed that different modalities of data in the news report correlate hierarchically. Traditional MSMO methods indistinguishably handle different modalities of data by learning a representation for the whole data, which is not directly adaptable to the heterogeneous contents and hierarchical correlation. In this paper, we propose a hierarchical cross-modality semantic correlation learning model (HCSCL) to learn the intra- and inter-modal correlation existing in the multimodal data. HCSCL adopts a graph network to encode the intra-modal correlation. Then, a hierarchical fusion framework is proposed to learn the hierarchical correlation between text and images. Furthermore, we construct a new dataset with relevant image annotation and image object label information to provide the supervision information for the learning procedure. Extensive experiments on the dataset show that HCSCL significantly outperforms the baseline methods in automatic summarization metrics and fine-grained diversity tests.