mlsanityData to 7 Oct 2026

TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities

arXiv 2212.06385 ↗First posted 13 Dec 2022cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Recently, the success of pre-training in text domain has been fully extended to vision, audio, and cross-modal scenarios. The proposed pre-training models of different modalities are showing a rising trend of homogeneity in their model structures, which brings the opportunity to implement different pre-training models within a uniform framework. In this paper, we present TencentPretrain, a toolkit supporting pre-training models of different modalities. The core feature of TencentPretrain is the modular design. The toolkit uniformly divides pre-training models into 5 components: embedding, encoder, target embedding, decoder, and target. As almost all of common modules are provided in each component, users can choose the desired modules from different components to build a complete pre-training model. The modular design enables users to efficiently reproduce existing pre-training models or build brand-new one. We test the toolkit on text, vision, and audio benchmarks and show that it can match the performance of the original implementations.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 39 of its 48 references that are in the map. First posted on arXiv 13 Dec 2022.

Builds on

39 of 48 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  12. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
27 more
  1. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2018Deep contextualized word representationscited by 2,285
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  8. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  9. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  10. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  11. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  12. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  13. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  14. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  15. 2017OpenNMT: Open-source Toolkit for Neural Machine Translationcited by 251
  16. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  17. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 230
  18. 2018An efficient framework for learning sentence representationscited by 197
  19. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  20. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  21. 2017Learned in Translation: Contextualized Word Vectorscited by 104
  22. 2020OpenAttack: An Open-source Textual Adversarial Attack Toolkitcited by 84
  23. 2020RealFormer: Transformer Likes Residual Attentioncited by 62
  24. 2019UER: An Open-Source Toolkit for Pre-training Modelscited by 61
  25. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  26. 2021PolyViT: Co-training Vision Transformers on Images, Videos and Audiocited by 58
  27. 2021ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generationcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 6; 2025: 3; 2026: 2.

19 papers.

Sort:

By area:

  1. 2023PersianLLaMA: Towards Building First Persian Large Language Modelcited by 14
  2. 2023Evaluating Large Language Models for Radiology Natural Language Processingcited by 12
  3. 2024CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Modelscited by 11
  4. 2023NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegiancited by 9
  5. 2023Weight-Inherited Distillation for Task-Agnostic BERT Compressioncited by 8
  6. 2024MockLLM: A Multi-Agent Behavior Collaboration Framework for Online Job Seeking and Recruitingcited by 8
  7. 2023Soulstyler: Using Large Language Model to Guide Image Style Transfer for Target Objectcited by 5
  8. 2023ZhuJiu: A Multi-dimensional, Multi-faceted Chinese Benchmark for Large Language Modelscited by 4
  9. 2023Create and Find Flatness: Building Flat Training Spaces in Advance for Continual Learningcited by 3
  10. 2024Dynamic data sampler for cross-language transfer learning in large language modelscited by 3
  11. 2023LMTuner: An user-friendly and highly-integrable Training Framework for fine-tuning Large Language Modelscited by 2
  12. 2025Language Models at the Syntax-Semantics Interface: A Case Study of the Long-Distance Binding of Chinese Reflexive zijicited by 2
7 more
  1. 2024LM-Combiner: A Contextual Rewriting Model for Chinese Grammatical Error Correctioncited by 1
  2. 2024Design as Desired: Utilizing Visual Question Answering for Multimodal Pre-trainingcited by 1
  3. 2025UrduLLaMA 1.0: Dataset Curation, Preprocessing, and Evaluation in Low-Resource Settingscited by 1
  4. 2024A Two-Phase Recall-and-Select Framework for Fast Model Selectioncited by 0
  5. 2025Quantifier Scope Interpretation in Language Learners and LLMscited by 0
  6. 2026Enhancing Target-Guided Proactive Dialogue Systems via Conversational Scenario Modeling and Intent-Keyword Bridgingcited by 0
  7. 2026Complexity-Guided Component-wise Initialization for Language Model Pretrainingcited by 0