mlsanityData to 14 Oct 2026

D2TV: Dual Knowledge Distillation and Target-oriented Vision Modeling for Many-to-Many Multimodal Summarization

arXiv 2305.12767 ↗First posted 22 May 2023cs.CV cs.AI cs.CLcited by 2

On the map: Knowledge distillation → Multimodal distillation

Abstract (arXiv)

Many-to-many multimodal summarization (M3S) task aims to generate summaries in any language with document inputs in any language and the corresponding image sequence, which essentially comprises multimodal monolingual summarization (MMS) and multimodal cross-lingual summarization (MXLS) tasks. Although much work has been devoted to either MMS or MXLS and has obtained increasing attention in recent years, little research pays attention to the M3S task. Besides, existing studies mainly focus on 1) utilizing MMS to enhance MXLS via knowledge distillation without considering the performance of MMS or 2) improving MMS models by filtering summary-unrelated visual features with implicit learning or explicitly complex training objectives. In this paper, we first introduce a general and practical task, i.e., M3S. Further, we propose a dual knowledge distillation and target-oriented vision modeling framework for the M3S task. Specifically, the dual knowledge distillation method guarantees that the knowledge of MMS and MXLS can be transferred to each other and thus mutually prompt both of them. To offer target-oriented visual features, a simple yet effective target-oriented contrastive objective is designed and responsible for discarding needless visual information. Extensive experiments on the many-to-many setting show the effectiveness of the proposed approach. Additionally, we will contribute a many-to-many multimodal summarization (M3Sum) dataset.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Multimodal distillation, one of 12 directions in Knowledge distillation. Builds on 27 of its 53 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

27 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021UniMS: A Unified Framework for Multimodal Summarization with Knowledge Distillation13 Sep

Other references · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  7. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  8. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  9. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 138
  10. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 132
  11. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  12. 2020VMSMO: Learning to Generate Multimodal Summary for Video-based News Articlescited by 44
14 more
  1. 2019Multimodal Abstractive Summarization for How2 Videoscited by 41
  2. 2022A Survey on Cross-Lingual Summarizationcited by 38
  3. 2021Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarizationcited by 32
  4. 2021Sequence Level Contrastive Learning for Text Summarizationcited by 31
  5. 2021Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillationcited by 22
  6. 2021Hierarchical Cross-Modality Semantic Correlation Learning Model for Multimodal Summarizationcited by 18
  7. 2022A Variational Hierarchical Model for Neural Cross-Lingual Summarizationcited by 17
  8. 2022MSCTD: A Multimodal Sentiment Chat Translation Datasetcited by 15
  9. 2023Towards Unifying Multi-Lingual and Cross-Lingual Summarizationcited by 15
  10. 2022Understanding Translationese in Cross-Lingual Summarizationcited by 14
  11. 2022Summary-Oriented Vision Modeling for Multimodal Abstractive Summarizationcited by 14
  12. 2023Towards Understanding and Improving Knowledge Distillation for Neural Machine Translationcited by 10
  13. 2020Emotional Conversation Generation with Heterogeneous Graph Neural Networkcited by 9
  14. 2023RC3: Regularized Contrastive Cross-lingual Cross-modal Pre-trainingcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2.

2 papers.

  1. 2023Cross-Lingual Knowledge Editing in Large Language Modelscited by 11
  2. 2023Snowman: A Million-scale Chinese Commonsense Knowledge Graph Distilled from Foundation Modelcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026MMLDSum-LLM: Multimodal Long-Document Summarization with Visual-Alignment and Keyword-Aware
  2. 2025Early Exit and Multi Stage Knowledge Distillation in VLMs for Video Summarization
  3. 2025Summarization of Multimodal Presentations with Vision-Language Models: Study of the Effect of Modalities and Structure
  4. 2025Integrating Video and Text: A Balanced Approach to Multimodal Summary Generation and Evaluation
  5. 2025M-DocSum: Do LVLMs Genuinely Comprehend Interleaved Image-Text in Document Summarization?
  6. 2024Realizing Video Summarization from the Path of Language-based Semantic Understanding
  7. 2024Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization
  8. 2026HAS: Highlight-guided Attention Steering for Multimodal LLM Video Summarization