# Machine-in-the-Loop Rewriting for Creative Image Captioning

arXiv: https://arxiv.org/abs/2111.04193 · HTML: https://mlsanity.com/paper/2111.04193/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 7 Nov 2021.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Reformulating Unsupervised Style Transfer as Paraphrase Generation](https://mlsanity.com/paper/2010.05700/) (2020)
- [On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation](https://mlsanity.com/paper/2005.03642/) (2020)
- [Enabling Language Models to Fill in the Blanks](https://mlsanity.com/paper/2005.05339/) (2020)
- [MaskGAN: Better Text Generation via Filling in the______](https://mlsanity.com/paper/1801.07736/) (2018)
- [PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction](https://mlsanity.com/paper/2010.13816/) (2020)
- [STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation](https://mlsanity.com/paper/2010.01717/) (2020)
- [MERMAID: Metaphor Generation with Symbolism and Discriminative Decoding](https://mlsanity.com/paper/2103.06779/) (2021)
- [Blank Language Models](https://mlsanity.com/paper/2002.03079/) (2020)
- [Wordcraft: a Human-AI Collaborative Editor for Story Writing](https://mlsanity.com/paper/2107.07430/) (2021)
- [GoodNewsEveryone: A Corpus of News Headlines Annotated with Emotions, Semantic Roles, and Reader Perception](https://mlsanity.com/paper/1912.03184/) (2019)
- [Iterative Edit-Based Unsupervised Sentence Simplification](https://mlsanity.com/paper/2006.09639/) (2020)
- [Langsmith: An Interactive Academic Text Revision System](https://mlsanity.com/paper/2010.04332/) (2020)
- [XL-Editor: Post-editing Sentences with XLNet](https://mlsanity.com/paper/1910.10479/) (2019)

## Built on by (most cited first, 11 of 11)

- [Does Writing with Language Models Reduce Content Diversity?](https://mlsanity.com/paper/2309.05196/) (2023)
- [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022)
- [A Design Space for Intelligent and Interactive Writing Assistants](https://mlsanity.com/paper/2403.14117/) (2024)
- [Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writing](https://mlsanity.com/paper/2210.13669/) (2022)
- [Read, Revise, Repeat: A System Demonstration for Human-in-the-loop Iterative Text Revision](https://mlsanity.com/paper/2204.03685/) (2022)
- [generAItor: Tree-in-the-Loop Text Generation for Language Model Explainability and Adaptation](https://mlsanity.com/paper/2403.07627/) (2024)
- [Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluation](https://mlsanity.com/paper/2406.07935/) (2024)
- [How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyond](https://mlsanity.com/paper/2501.05714/) (2025)
- [Conveying the Predicted Future to Users: A Case Study of Story Plot Prediction](https://mlsanity.com/paper/2302.09122/) (2023)
- [Understanding Revision Behavior in Adaptive Writing Support Systems for Education](https://mlsanity.com/paper/2306.10304/) (2023)
- [Towards Cognitive Process-Aware Proactive Writing Support](https://mlsanity.com/paper/2608.30424/) (2026)

## Abstract (arXiv)

Machine-in-the-loop writing aims to enable humans to collaborate with models to complete their writing tasks more effectively. Prior work has found that providing humans a machine-written draft or sentence-level continuations has limited success since the generated text tends to deviate from humans' intention. To allow the user to retain control over the content, we train a rewriting model that, when prompted, modifies specified spans of text within the user's original draft to introduce descriptive and figurative elements locally in the text. We evaluate the model on its ability to collaborate with humans on the task of creative image captioning. On a user study through Amazon Mechanical Turk, our model is rated to be more helpful than a baseline infilling language model. In addition, third-party evaluation shows that users write more descriptive and figurative captions when collaborating with our model compared to completing the task alone.