mlsanityData to 7 Oct 2026

Putting Humans in the Image Captioning Loop

arXiv 2306.03476 ↗First posted 6 Jun 2023cs.CL cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Image Captioning (IC) models can highly benefit from human feedback in the training process, especially in cases where data is limited. We present work-in-progress on adapting an IC system to integrate human feedback, with the goal to make it easily adaptable to user-specific data. Our approach builds on a base IC model pre-trained on the MS COCO dataset, which generates captions for unseen images. The user will then be able to offer feedback on the image and the generated/predicted caption, which will be augmented to create additional training instances for the adaptation of the model. The additional instances are integrated into the model using step-wise updates, and a sparse memory replay component is used to avoid catastrophic forgetting. We hope that this approach, while leading to improved results, will also result in customizable IC models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

21 of 31 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  6. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  8. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  9. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  10. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  11. 2018Albumentations: fast and flexible image augmentationscited by 558
  12. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
9 more
  1. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  2. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  3. 2019Episodic Memory in Lifelong Language Learningcited by 188
  4. 2020Captioning Images Taken by People Who Are Blindcited by 108
  5. 2020RATT: Recurrent Attention to Transient Tasks for Continual Image Captioningcited by 32
  6. 2018Learning to Caption Images through a Lifetime by Asking Questionscited by 11
  7. 2019ContCap: A scalable framework for continual image captioningcited by 11
  8. 2017Teaching Machines to Describe Images via Natural Language Feedbackcited by 8
  9. 2022Interactive Machine Learning for Image Captioningcited by 3

Built on by

Not cited yet.