mlsanityData to 7 Oct 2026

Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision

arXiv 2010.06775 ↗First posted 14 Oct 2020cs.CL cs.AI cs.CVcited by 93

Not on the map. Why?

Abstract (arXiv)

Humans learn language by listening, speaking, writing, reading, and also, via interaction with the multimodal real world. Existing language pre-training frameworks show the effectiveness of text-only self-supervision while we explore the idea of a visually-supervised language model in this paper. We find that the main reason hindering this exploration is the large divergence in magnitude and distributions between the visually-grounded language datasets and pure-language corpora. Therefore, we develop a technique named "vokenization" that extrapolates multimodal alignments to language-only data by contextually mapping language tokens to their related images (which we call "vokens"). The "vokenizer" is trained on relatively small image captioning datasets and we then apply it to generate vokens for large language corpora. Trained with these contextually generated vokens, our visually-supervised language models show consistent improvements over self-supervised alternatives on multiple pure-language tasks such as GLUE, SQuAD, and SWAG. Code and pre-trained models publicly available at https://github.com/airsplay/vokenization

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 33; 2023: 21; 2024: 12; 2025: 8; 2026: 3.

93 papers.

Sort:

By area:

  1. 2021Transformers in Vision: A Surveycited by 1,009
  2. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  3. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  4. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  5. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  6. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  7. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  8. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  9. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  10. 2023Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillingscited by 68
  11. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  12. 2023On Uni-Modal Feature Learning in Supervised Multi-Modal Learningcited by 53
38 more
  1. 2023VIP5: Towards Multimodal Foundation Models for Recommendationcited by 45
  2. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  3. 2022Visual Commonsense in Pretrained Unimodal and Multimodal Modelscited by 39
  4. 2021The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Colorcited by 32
  5. 2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
  6. 2022Visualize Before You Write: Imagination-Guided Open-Ended Text Generationcited by 30
  7. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  8. 2022VALHALLA: Visual Hallucination for Machine Translationcited by 27
  9. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  10. 2021Does Vision-and-Language Pretraining Improve Lexical Grounding?cited by 25
  11. 2022Unsupervised Vision-and-Language Pre-training via Retrieval-based Multi-Granular Alignmentcited by 23
  12. 2021VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfercited by 22
  13. 2022MCSE: Multimodal Contrastive Learning of Sentence Embeddingscited by 22
  14. 2021Maria: A Visual Experience Powered Conversational Agentcited by 19
  15. 2022UNIMO-2: End-to-End Unified Vision-Language Grounded Learningcited by 19
  16. 2022Imagination-Augmented Natural Language Understandingcited by 19
  17. 2023Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understandingcited by 19
  18. 2023Can Language Models Understand Physical Concepts?cited by 19
  19. 2022Visually-Augmented Language Modelingcited by 18
  20. 2022Z-LaVI: Zero-Shot Language Solver Fueled by Visual Imaginationcited by 18
  21. 2023Putting Natural in Natural Language Processingcited by 17
  22. 2023MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokenscited by 15
  23. 2023Universal Multimodal Representation for Language Understandingcited by 14
  24. 2024KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learningcited by 13
  25. 2025VIST-GPT: Ushering in the Era of Visual Storytelling with LLMs?cited by 11
  26. 2022Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-modal Knowledge Transfercited by 10
  27. 2023World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Modelscited by 10
  28. 2024Vision-Language Navigation with Embodied Intelligence: A Surveycited by 9
  29. 2021Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Trainingcited by 8
  30. 2021Visual Grounding Strategies for Text-Only Natural Language Processingcited by 7
  31. 2021Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Researchcited by 7
  32. 2022Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attentioncited by 7
  33. 2023Learning to Imagine: Visually-Augmented Natural Language Generationcited by 7
  34. 2022Text-Derived Knowledge Helps Vision: A Simple Cross-modal Distillation for Video-based Action Anticipationcited by 6
  35. 2024Lexicon-Level Contrastive Visual-Grounding Improves Language Modelingcited by 6
  36. 2022Language with Vision: a Study on Grounded Word and Sentence Embeddingscited by 5
  37. 2022Visually-augmented pretrained language models for NLP tasks without imagescited by 5
  38. 2022How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input?cited by 4

50 most cited of 93 All 93 (JSON)