Sample Efficient Multimodal Semantic Augmentation for Incremental Summarization
Not on the map. Why?
Abstract (arXiv)
In this work, we develop a prompting approach for incremental summarization of task videos. We develop a sample-efficient few-shot approach for extracting semantic concepts as an intermediate step. We leverage an existing model for extracting the concepts from the images and extend it to videos and introduce a clustering and querying approach for sample efficiency, motivated by the recent advances in perceiver-based architectures. Our work provides further evidence that an approach with richer input context with relevant entities and actions from the videos and using these as prompts could enhance the summaries generated by the model. We show the results on a relevant dataset and discuss possible directions for the work.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 49 references that are in the map. First posted on arXiv 8 Mar 2023.
Builds on
32 of 49 references.
References · 32
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
- 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
- 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
- 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
- 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
- 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
- 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
20 more
- 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
- 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
- 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
- 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
- 2019Large-scale weakly-supervised pre-training for video action recognitioncited by 130
- 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
- 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
- 2020Multi-Fact Correction in Abstractive Text Summarizationcited by 66
- 2017Learning Robust Visual-Semantic Embeddingscited by 56
- 2017Graph-based Neural Multi-Document Summarizationcited by 54
- 2020Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Modelscited by 44
- 2019Multimodal Abstractive Summarization for How2 Videoscited by 41
- 2019Topic-aware Pointer-Generator Networks for Summarizing Spoken Conversationscited by 41
- 2019Earlier Isn't Always Better: Sub-aspect Analysis on Corpus and System Biases in Summarizationcited by 28
- 2018Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generationcited by 25
- 2023Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipationcited by 21
- 2016Learning Joint Representations of Videos and Sentences with Web Image Searchcited by 17
- 2022PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarizationcited by 14
- 2019Efficient Adaptation of Pretrained Transformers for Abstractive Summarizationcited by 13
- 2022Improving abstractive summarization with energy-based re-rankingcited by 3
Built on by
Not cited yet.