# Fusing Multimodal Signals on Hyper-complex Space for Extreme Abstractive Text Summarization (TL;DR) of Scientific Contents arXiv: https://arxiv.org/abs/2306.13968 ยท HTML: https://mlsanity.com/paper/2306.13968/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 39 of its 73 references that are in the map. - First posted on arXiv 24 Jun 2023. ## Builds on (most cited first, 30 of 39) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014) - [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020) - [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017) - [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017) - [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019) - [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015) - [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016) - [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017) - [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019) - [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016) - [Bottom-Up Abstractive Summarization](https://mlsanity.com/paper/1808.10792/) (2018) - [How2: A Large-scale Dataset for Multimodal Language Understanding](https://mlsanity.com/paper/1811.00347/) (2018) - [TLDR: Extreme Summarization of Scientific Documents](https://mlsanity.com/paper/2004.15011/) (2020) - [Fine-tune BERT for Extractive Summarization](https://mlsanity.com/paper/1903.10318/) (2019) - [Neural Summarization by Extracting Sentences and Words](https://mlsanity.com/paper/1603.07252/) (2016) - [Multi-modal Dense Video Captioning](https://mlsanity.com/paper/2003.07758/) (2020) - [Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameters](https://mlsanity.com/paper/2102.08597/) (2021) - [Sentence Centrality Revisited for Unsupervised Summarization](https://mlsanity.com/paper/1906.03508/) (2019) - [Adapting the Neural Encoder-Decoder Framework from Single to Multi-Document Summarization](https://mlsanity.com/paper/1808.06218/) (2018) - [Multimodal Abstractive Summarization for How2 Videos](https://mlsanity.com/paper/1906.07901/) (2019) - [Attention Strategies for Multi-Source Sequence-to-Sequence Learning](https://mlsanity.com/paper/1704.06567/) (2017) - [UniMS: A Unified Framework for Multimodal Summarization with Knowledge Distillation](https://mlsanity.com/paper/2109.05812/) (2021) - [See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization](https://mlsanity.com/paper/2105.09601/) (2021) - [Title Generation for User Generated Videos](https://mlsanity.com/paper/1608.07068/) (2016) - [Structure-Infused Copy Mechanisms for Abstractive Summarization](https://mlsanity.com/paper/1806.05658/) (2018) - [CiteSum: Citation Text-guided Scientific Extreme Summarization and Domain Adaptation with Limited Supervision](https://mlsanity.com/paper/2205.06207/) (2022) ## Built on by (most cited first, 7 of 7) - [Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarization](https://mlsanity.com/paper/2409.14907/) (2024) - [PerSEval: Assessing Personalization in Text Summarizers](https://mlsanity.com/paper/2407.00453/) (2024) - [Model Editing with Graph-Based External Memory](https://mlsanity.com/paper/2505.18343/) (2025) - [Continually Self-Improving Language Models for Bariatric Surgery Question--Answering](https://mlsanity.com/paper/2505.16102/) (2025) - [Exploiting Representation Bias for Data Distillation in Abstractive Text Summarization](https://mlsanity.com/paper/2312.06022/) (2023) - [A Structured Literature Review on Traditional Approaches in Current Natural Language Processing](https://mlsanity.com/paper/2505.12970/) (2025) - [Not too long do read: Evaluating LLM-generated extreme scientific summaries](https://mlsanity.com/paper/2512.23206/) (2025) ## Abstract (arXiv) The realm of scientific text summarization has experienced remarkable progress due to the availability of annotated brief summaries and ample data. However, the utilization of multiple input modalities, such as videos and audio, has yet to be thoroughly explored. At present, scientific multimodal-input-based text summarization systems tend to employ longer target summaries like abstracts, leading to an underwhelming performance in the task of text summarization. In this paper, we deal with a novel task of extreme abstractive text summarization (aka TL;DR generation) by leveraging multiple input modalities. To this end, we introduce mTLDR, a first-of-its-kind dataset for the aforementioned task, comprising videos, audio, and text, along with both author-composed summaries and expert-annotated summaries. The mTLDR dataset accompanies a total of 4,182 instances collected from various academic conference proceedings, such as ICLR, ACL, and CVPR. Subsequently, we present mTLDRgen, an encoder-decoder-based model that employs a novel dual-fused hyper-complex Transformer combined with a Wasserstein Riemannian Encoder Transformer, to dexterously capture the intricacies between different modalities in a hyper-complex latent geometric space. The hyper-complex Transformer captures the intrinsic properties between the modalities, while the Wasserstein Riemannian Encoder Transformer captures the latent structure of the modalities in the latent space geometry, thereby enabling the model to produce diverse sentences. mTLDRgen outperforms 20 baselines on mTLDR as well as another non-scientific dataset (How2) across three Rouge-based evaluation measures. Furthermore, based on the qualitative metrics, BERTScore and FEQA, and human evaluations, we demonstrate that the summaries generated by mTLDRgen are fluent and congruent to the original source material.