mlsanityData to 7 Oct 2026

Meta-Transformer: A Unified Framework for Multimodal Learning

arXiv 2307.10802 ↗First posted 20 Jul 2023cs.CV cs.AI cs.CLcited by 136

Not on the map. Why?

Abstract (arXiv)

Multimodal learning aims to build models that can process and relate information from multiple modalities. Despite years of development in this field, it still remains challenging to design a unified network for processing various modalities (textite.g. natural language, 2D images, 3D point clouds, audio, video, time series, tabular data) due to the inherent gaps among them. In this work, we propose a framework, named Meta-Transformer, that leverages a frozen encoder to perform multimodal perception without any paired multimodal training data. In Meta-Transformer, the raw input data from various modalities are mapped into a shared token space, allowing a subsequent encoder with frozen parameters to extract high-level semantic features of the input data. Composed of three main components: a unified data tokenizer, a modality-shared encoder, and task-specific heads for downstream tasks, Meta-Transformer is the first framework to perform unified learning across 12 modalities with unpaired data. Experiments on different benchmarks reveal that Meta-Transformer can handle a wide range of tasks including fundamental perception (text, image, point cloud, audio, video), practical application (X-Ray, infrared, hyperspectral, and IMU), and data mining (graph, tabular, and time-series). Meta-Transformer indicates a promising future for developing unified multimodal intelligence with transformers. Code will be available at https://github.com/invictus717/MetaTransformer

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). Builds on 93 of its 123 references that are in the map. First posted on arXiv 20 Jul 2023.

Builds on

93 of 123 references.

References · 93

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  12. 2020End-to-End Object Detection with Transformerscited by 7,384
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  2. 2017Mask R-CNNcited by 7,259
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2018How Powerful are Graph Neural Networks?cited by 4,355
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2014Going Deeper with Convolutionscited by 4,231
  9. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  10. 2022A ConvNet for the 2020scited by 3,545
  11. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  12. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  13. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  14. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  15. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  16. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  17. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  18. 2020Reformer: The Efficient Transformercited by 1,658
  19. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  20. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  21. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  22. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  23. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  24. 2020Point Transformercited by 1,462
  25. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  26. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  27. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  28. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  29. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  30. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  31. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  32. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  33. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,030
  34. 2021Scaling Vision Transformerscited by 993
  35. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  36. 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 869
  37. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  38. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809

50 most cited of 93 All 93 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 32; 2024: 52; 2025: 41; 2026: 11.

136 papers.

Sort:

By area:

  1. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  2. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  3. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  4. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 231
  5. 2023Language is All a Graph Needscited by 219
  6. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  7. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  8. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  9. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 101
  10. 2024Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observationcited by 97
  11. 2024A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challengescited by 86
  12. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
38 more
  1. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 69
  2. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  3. 2023UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognitioncited by 64
  4. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 61
  5. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  6. 20244M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalitiescited by 52
  7. 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49
  8. 2023SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-trainingcited by 41
  9. 2024Autoregressive Models in Vision: A Surveycited by 39
  10. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 38
  11. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  12. 2023The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Modelscited by 34
  13. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learningcited by 30
  14. 2023ViT-Lens: Towards Omni-modal Representationscited by 29
  15. 2024Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalitiescited by 29
  16. 2024Safety Alignment for Vision Language Modelscited by 27
  17. 2025SkySense V2: A Unified Foundation Model for Multi-modal Remote Sensingcited by 26
  18. 2024MIO: A Foundation Model on Multimodal Tokenscited by 24
  19. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  20. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  21. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  22. 2025NTIRE 2025 Challenge on Day and Night Raindrop Removal for Dual-Focused Images: Methods and Resultscited by 22
  23. 2024Transformer for Object Re-Identification: A Surveycited by 21
  24. 2024Trajectory Flow Matching with Applications to Clinical Time Series Modelingcited by 21
  25. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  26. 2023T-Rex: Counting by Visual Promptingcited by 20
  27. 2024OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of Allcited by 20
  28. 2024XTrack: Multimodal Training Boosts RGB-X Video Object Trackerscited by 20
  29. 2023Pre-training on Synthetic Driving Data for Trajectory Predictioncited by 18
  30. 2023RED-DOT: Multimodal Fact-checking via Relevant Evidence Detectioncited by 18
  31. 2024All in One Framework for Multimodal Re-identification in the Wildcited by 18
  32. 2024IoT-LLM: a framework for enhancing Large Language Model reasoning from real-world sensor datacited by 18
  33. 2024RELI11D: A Comprehensive Multimodal Human Motion Dataset and Methodcited by 16
  34. 2024Large Language Models for Mobility Analysis in Transportation Systems: A Survey on Forecasting Taskscited by 15
  35. 2024Vision Search Assistant: Empower Vision-Language Models as Multimodal Search Enginescited by 14
  36. 2023When Geoscience Meets Foundation Models: Towards General Geoscience Artificial Intelligence Systemcited by 13
  37. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  38. 2025OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learningcited by 13

50 most cited of 136 All 136 (JSON)