mlsanityData to 7 Oct 2026

Survey: Transformer-based Models in Data Modality Conversion

arXiv 2408.04723 ↗First posted 8 Aug 2024eess.IV cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Transformers have made significant strides across various artificial intelligence domains, including natural language processing, computer vision, and audio processing. This success has naturally garnered considerable interest from both academic and industry researchers. Consequently, numerous Transformer variants (often referred to as X-formers) have been developed for these fields. However, a thorough and systematic review of these modality-specific conversions remains lacking. Modality Conversion involves the transformation of data from one form of representation to another, mimicking the way humans integrate and interpret sensory information. This paper provides a comprehensive review of transformer-based models applied to the primary modalities of text, vision, and speech, discussing their architectures, conversion methodologies, and applications. By synthesizing the literature on modality conversion, this survey aims to underline the versatility and scalability of transformers in advancing AI-driven content generation and understanding.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 116 of its 179 references that are in the map. First posted on arXiv 8 Aug 2024.

Builds on

116 of 179 references.

References · 116

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  11. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  13. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  14. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  15. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  16. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  17. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  18. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  19. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  20. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  21. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  22. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  23. 2021Transformers in Vision: A Surveycited by 1,009
  24. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  25. 2020Pre-Trained Image Processing Transformercited by 891
  26. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  27. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  28. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  29. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  30. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  31. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  32. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  33. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  34. 2018CoQA: A Conversational Question Answering Challengecited by 674
  35. 2020A Survey on Visual Transformercited by 653
  36. 2015Listen, Attend and Spellcited by 542
  37. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  38. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496

50 most cited of 116 All 116 (JSON)

Built on by

Not cited yet.