mlsanityData to 7 Oct 2026

Affective Feedback Synthesis Towards Multimodal Text and Image Data

arXiv 2203.12692 ↗First posted 23 Mar 2022cs.MM cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

In this paper, we have defined a novel task of affective feedback synthesis that deals with generating feedback for input text & corresponding image in a similar way as humans respond towards the multimodal data. A feedback synthesis system has been proposed and trained using ground-truth human comments along with image-text input. We have also constructed a large-scale dataset consisting of image, text, Twitter user comments, and the number of likes for the comments by crawling the news articles through Twitter feeds. The proposed system extracts textual features using a transformer-based textual encoder while the visual features have been extracted using a Faster region-based convolutional neural networks model. The textual and visual features have been concatenated to construct the multimodal features using which the decoder synthesizes the feedback. We have compared the results of the proposed system with the baseline models using quantitative and qualitative measures. The generated feedbacks have been analyzed using automatic and human evaluation. They have been found to be semantically similar to the ground-truth comments and relevant to the given text-image input.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 33 of its 70 references that are in the map. First posted on arXiv 23 Mar 2022.

Builds on

33 of 70 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  6. 2015Fast R-CNNcited by 2,930
  7. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  10. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  11. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  12. 2017Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoderscited by 222
21 more
  1. 2016Visual Storytellingcited by 176
  2. 2017Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memorycited by 167
  3. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  4. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  5. 2019Audio-Visual Scene-Aware Dialogcited by 122
  6. 2017MojiTalk: Generating Emotional Responses at Scalecited by 64
  7. 2018DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encodercited by 44
  8. 2020VMSMO: Learning to Generate Multimodal Summary for Video-based News Articlescited by 44
  9. 2018Response Generation by Context-aware Prototype Editingcited by 40
  10. 2018End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Featurescited by 37
  11. 2018Recursive Visual Attention in Visual Dialogcited by 34
  12. 2019DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialoguecited by 30
  13. 2019What Makes A Good Story? Designing Composite Rewards for Visual Storytellingcited by 29
  14. 2018Better Conversations by Modeling,Filtering,and Optimizing for Coherence and Diversitycited by 27
  15. 2019Dual Attention Networks for Visual Reference Resolution in Visual Dialogcited by 25
  16. 2021Multimodal Emotion Recognition with High-level Speech and Text Featurescited by 20
  17. 2018Multimodal Sentiment Analysis To Explore the Structure of Emotionscited by 19
  18. 2020From Standard Summarization to New Tasks and Beyond: Summarization with Manifold Informationcited by 16
  19. 2019DMRM: A Dual-channel Multi-hop Reasoning Model for Visual Dialogcited by 14
  20. 2017Neural Extractive Summarization with Side Informationcited by 6
  21. 2020Multi-View Attention Network for Visual Dialogcited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Synthesizing Sentiment-Controlled Feedback For Multimodal Text and Image Datacited by 0