# D2TV: Dual Knowledge Distillation and Target-oriented Vision Modeling for Many-to-Many Multimodal Summarization

arXiv: https://arxiv.org/abs/2305.12767 · HTML: https://mlsanity.com/paper/2305.12767/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 27 of its 53 references that are in the map.
- First posted on arXiv 22 May 2023.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Unifying Vision-and-Language Tasks via Text Generation](https://mlsanity.com/paper/2102.02779/) (2021)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [How2: A Large-scale Dataset for Multimodal Language Understanding](https://mlsanity.com/paper/1811.00347/) (2018)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2109.09209/) (2021)
- [VMSMO: Learning to Generate Multimodal Summary for Video-based News Articles](https://mlsanity.com/paper/2010.05406/) (2020)
- [Multimodal Abstractive Summarization for How2 Videos](https://mlsanity.com/paper/1906.07901/) (2019)
- [A Survey on Cross-Lingual Summarization](https://mlsanity.com/paper/2203.12515/) (2022)
- [Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization](https://mlsanity.com/paper/2109.02401/) (2021)
- [Sequence Level Contrastive Learning for Text Summarization](https://mlsanity.com/paper/2109.03481/) (2021)
- [UniMS: A Unified Framework for Multimodal Summarization with Knowledge Distillation](https://mlsanity.com/paper/2109.05812/) (2021)
- [Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillation](https://mlsanity.com/paper/2112.03473/) (2021)
- [Hierarchical Cross-Modality Semantic Correlation Learning Model for Multimodal Summarization](https://mlsanity.com/paper/2112.12072/) (2021)
- [A Variational Hierarchical Model for Neural Cross-Lingual Summarization](https://mlsanity.com/paper/2203.03820/) (2022)
- [MSCTD: A Multimodal Sentiment Chat Translation Dataset](https://mlsanity.com/paper/2202.13645/) (2022)
- [Towards Unifying Multi-Lingual and Cross-Lingual Summarization](https://mlsanity.com/paper/2305.09220/) (2023)
- [Understanding Translationese in Cross-Lingual Summarization](https://mlsanity.com/paper/2212.07220/) (2022)
- [Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization](https://mlsanity.com/paper/2212.07672/) (2022)
- [Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2305.08096/) (2023)
- [Emotional Conversation Generation with Heterogeneous Graph Neural Network](https://mlsanity.com/paper/2012.04882/) (2020)
- [RC3: Regularized Contrastive Cross-lingual Cross-modal Pre-training](https://mlsanity.com/paper/2305.07927/) (2023)

## Built on by (most cited first, 2 of 2)

- [Cross-Lingual Knowledge Editing in Large Language Models](https://mlsanity.com/paper/2309.08952/) (2023)
- [Snowman: A Million-scale Chinese Commonsense Knowledge Graph Distilled from Foundation Model](https://mlsanity.com/paper/2306.10241/) (2023)

## Abstract (arXiv)

Many-to-many multimodal summarization (M3S) task aims to generate summaries in any language with document inputs in any language and the corresponding image sequence, which essentially comprises multimodal monolingual summarization (MMS) and multimodal cross-lingual summarization (MXLS) tasks. Although much work has been devoted to either MMS or MXLS and has obtained increasing attention in recent years, little research pays attention to the M3S task. Besides, existing studies mainly focus on 1) utilizing MMS to enhance MXLS via knowledge distillation without considering the performance of MMS or 2) improving MMS models by filtering summary-unrelated visual features with implicit learning or explicitly complex training objectives. In this paper, we first introduce a general and practical task, i.e., M3S. Further, we propose a dual knowledge distillation and target-oriented vision modeling framework for the M3S task. Specifically, the dual knowledge distillation method guarantees that the knowledge of MMS and MXLS can be transferred to each other and thus mutually prompt both of them. To offer target-oriented visual features, a simple yet effective target-oriented contrastive objective is designed and responsible for discarding needless visual information. Extensive experiments on the many-to-many setting show the effectiveness of the proposed approach. Additionally, we will contribute a many-to-many multimodal summarization (M3Sum) dataset.