mlsanityData to 14 Oct 2026

TVLT: Textless Vision-Language Transformer

arXiv 2209.14156 ↗First posted 28 Sep 2022cs.CV cs.AI cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

In this work, we present the Textless Vision-Language Transformer (TVLT), where homogeneous transformer blocks take raw visual and audio inputs for vision-and-language representation learning with minimal modality-specific design, and do not use text-specific modules such as tokenization or automatic speech recognition (ASR). TVLT is trained by reconstructing masked patches of continuous video frames and audio spectrograms (masked autoencoding) and contrastive modeling to align video and audio. TVLT attains performance comparable to its text-based counterpart on various multimodal tasks, such as visual question answering, image retrieval, video retrieval, and multimodal sentiment analysis, with 28x faster inference speed and only 1/3 of the parameters. Our findings suggest the possibility of learning compact and efficient visual-linguistic representations from low-level visual and audio signals without assuming the prior existence of text. Our code and checkpoints are available at: https://github.com/zinengtang/TVLT

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 71 of its 99 references that are in the map. First posted on arXiv 28 Sep 2022.

Builds on

71 of 99 references.

References · 71

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  2. 2015VQA: Visual Question Answeringcited by 2,506
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2018Deep contextualized word representationscited by 2,290
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  8. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  11. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  13. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  14. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  15. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  16. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  17. 2021AST: Audio Spectrogram Transformercited by 643
  18. 2019Green AIcited by 627
  19. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  20. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  21. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  22. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  23. 2017Look, Listen and Learncited by 458
  24. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  25. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  26. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  27. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  28. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 347
  29. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  30. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  31. 2018Deep Audio-Visual Speech Recognitioncited by 322
  32. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  33. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  34. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  35. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 249
  36. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  37. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 228
  38. 2019Cross-task weakly supervised learning from instructional videoscited by 199

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 9; 2024: 5; 2025: 5.

22 papers.

Sort:

By area:

  1. 2023Any-to-Any Generation via Composable Diffusioncited by 204
  2. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  3. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  4. 2022MAViL: Masked Audio-Video Learnerscited by 59
  5. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 55
  6. 2023VidChapters-7M: Video Chapters at Scalecited by 39
  7. 2024CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusioncited by 17
  8. 2024Siamese Vision Transformers are Scalable Audio-visual Learnerscited by 14
  9. 2023BirdSAT: Cross-View Contrastive Masked Autoencoders for Bird Species Classification and Mappingcited by 12
  10. 2025Multi-modal brain encoding models for multi-modal stimulicited by 8
  11. 2023MOSEL: Inference Serving Using Dynamic Modality Selectioncited by 5
  12. 2025Brain-Language Model Alignment: Insights into the Platonic Hypothesis and Intermediate-Layer Advantagecited by 5
10 more
  1. 2024Comment-aided Video-Language Alignment via Contrastive Pre-training for Short-form Video Humor Detectioncited by 4
  2. 2023Exploring the Role of Audio in Video Captioningcited by 3
  3. 2023Modality Plug-and-Play: Elastic Modality Adaptation in Multimodal LLMs for Embodied AIcited by 2
  4. 2024Unified Video-Language Pre-training with Synchronized Audiocited by 2
  5. 2023STELLA: Continual Audio-Video Pre-training with Spatio-Temporal Localized Alignmentcited by 1
  6. 2025Spoken question answering for visual queriescited by 1
  7. 2025The One Where They Brain-Tune for Social Cognition: Multi-Modal Brain-Tuning on Friendscited by 1
  8. 2023A Unified Transformer-based Network for multimodal Emotion Recognitioncited by 0
  9. 2024Multimodal Sentiment Analysis based on Video and Audio Inputscited by 0
  10. 2025Seeing What You Say: Expressive Image Generation from Speechcited by 0