mlsanityData to 7 Oct 2026

Visualize Before You Write: Imagination-Guided Open-Ended Text Generation

arXiv 2210.03765 ↗First posted 7 Oct 2022cs.CL cs.AIcited by 30

Not on the map. Why?

Abstract (arXiv)

Recent advances in text-to-image synthesis make it possible to visualize machine imaginations for a given context. On the other hand, when generating text, human writers are gifted at creative visualization, which enhances their writings by forming imaginations as blueprints before putting down the stories in words. Inspired by such a cognitive process, we ask the natural question of whether we can endow machines with the same ability to utilize visual information and construct a general picture of the context to guide text generation. In this work, we propose iNLG that uses machine-generated images to guide language models in open-ended text generation. The experiments and analyses demonstrate the effectiveness of iNLG on open-ended text generation tasks, including text completion, story generation, and concept-to-text generation in both few-shot and full-data scenarios. Both automatic metrics and human evaluations verify that the text snippets generated by our iNLG are coherent and informative while displaying minor degeneration.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 60 of its 86 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

60 of 86 references.

References · 60

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
38 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2018Hierarchical Neural Story Generationcited by 1,182
  9. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  10. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  11. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  12. 2016Language Modeling with Gated Convolutional Networkscited by 688
  13. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  14. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  15. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  16. 2017Convolutional Sequence to Sequence Learningcited by 550
  17. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  18. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  19. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  20. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  21. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  22. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  23. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  24. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  25. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  26. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  27. 2018Plan-And-Write: Towards Better Automatic Storytellingcited by 232
  28. 2022A Contrastive Framework for Neural Text Generationcited by 224
  29. 2016Visual Storytellingcited by 176
  30. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  31. 2020A Knowledge-Enhanced Pretraining Model for Commonsense Story Generationcited by 116
  32. 2020KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoningcited by 106
  33. 2020Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervisioncited by 93
  34. 2020Content Planning for Neural Story Generation with Aristotelian Rescoringcited by 92
  35. 2019Do Massively Pretrained Language Models Make Better Storytellers?cited by 90
  36. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89
  37. 2017Cold Fusion: Training Seq2Seq Models Together with Language Modelscited by 87
  38. 2018Story Ending Generation with Incremental Encoding and Commonsense Knowledgecited by 60

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 15; 2024: 10; 2025: 3; 2026: 1.

30 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2023LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluationcited by 85
  3. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  4. 2024A Design Space for Intelligent and Interactive Writing Assistantscited by 76
  5. 2023Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillingscited by 68
  6. 2023Context-I2W: Mapping Images to Context-dependent Words for Accurate Zero-Shot Composed Image Retrievalcited by 52
  7. 2023Multimodal Procedural Planning via Dual Text-Image Promptingcited by 40
  8. 2024List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMscited by 32
  9. 2023LMEye: An Interactive Perception Network for Large Language Modelscited by 27
  10. 2024On the Evaluation of Machine-Generated Reportscited by 23
  11. 2024MORE: Multi-mOdal REtrieval Augmented Generative Commonsense Reasoningcited by 12
  12. 2025Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Datacited by 12
18 more
  1. 2025Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generationcited by 12
  2. 2023A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Cluescited by 11
  3. 2023Training Multimedia Event Extraction With Generated Images and Captionscited by 11
  4. 2023Learning to Imagine: Visually-Augmented Natural Language Generationcited by 7
  5. 2023Prefix-diffusion: A Lightweight Diffusion Model for Diverse Image Captioningcited by 7
  6. 2022Visually-augmented pretrained language models for NLP tasks without imagescited by 5
  7. 2023Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMscited by 4
  8. 2024A Multimodal In-Context Tuning Approach for E-Commerce Product Description Generationcited by 4
  9. 2024Cognitive Visual-Language Mapper: Advancing Multimodal Comprehension with Enhanced Visual Knowledge Alignmentcited by 3
  10. 2024Denoise-I2W: Mapping Images to Denoising Words for Accurate Zero-Shot Composed Image Retrievalcited by 3
  11. 2023"Nothing Abnormal": Disambiguating Medical Reports via Contrastive Knowledge Infusioncited by 2
  12. 2023Training on Synthetic Data Beats Real Data in Multimodal Relation Extractioncited by 2
  13. 2023In BLOOM: Creativity and Affinity in Artificial Lyrics and Artcited by 1
  14. 2024WinoViz: Probing Visual Properties of Objects Under Different Statescited by 1
  15. 2024Context-aware Visual Storytelling with Visual Prefix Tuning and Contrastive Learningcited by 1
  16. 2024Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Groundingcited by 1
  17. 2025HERAKLES: Hierarchical Skill Compilation for Open-ended LLM Agentscited by 0
  18. 2026SynMulti: Synthetic-to-Real Learning for Multimodal Video Understandingcited by 0