mlsanityData to 7 Oct 2026

Text-centric Alignment for Multi-Modality Learning

arXiv 2402.08086 ↗First posted 12 Feb 2024cs.LG cs.CL cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

This research paper addresses the challenge of modality mismatch in multimodal learning, where the modalities available during inference differ from those available at training. We propose the Text-centric Alignment for Multi-Modality Learning (TAMML) approach, an innovative method that utilizes Large Language Models (LLMs) with in-context learning and foundation models to enhance the generalizability of multimodal systems under these conditions. By leveraging the unique properties of text as a unified semantic space, TAMML demonstrates significant improvements in handling unseen, diverse, and unpredictable modality combinations. TAMML not only adapts to varying modalities but also maintains robust performance, showcasing the potential of foundation models in overcoming the limitations of traditional fixed-modality frameworks in embedding representations. This study contributes to the field by offering a flexible, effective solution for real-world applications where modality availability is dynamic and uncertain.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 20 of its 46 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

20 of 46 references.

References · 20

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2020Denoising Diffusion Implicit Modelscited by 9,655
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  7. 2017Adversarial Discriminative Domain Adaptationcited by 1,309
  8. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  9. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  10. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  11. 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
  12. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
8 more
  1. 2015Label-Embedding for Image Classificationcited by 164
  2. 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 125
  3. 2016Neural Photo Editing with Introspective Adversarial Networkscited by 109
  4. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  5. 2021SemanticStyleGAN: Learning Compositional Generative Priors for Controllable Image Synthesis and Editingcited by 58
  6. 2017Generating Visual Representations for Zero-Shot Classificationcited by 37
  7. 2017Zero-Shot Learning via Class-Conditioned Deep Generative Modelscited by 21
  8. 2023Zero-shot-Learning Cross-Modality Data Translation Through Mutual Information Guided Stochastic Diffusioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 4; 2026: 1.

12 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  3. 2024Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruningcited by 19
  4. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  5. 2025Multimodal Emotion Recognition in Conversations: A Survey of Methods, Trends, Challenges and Prospectscited by 10
  6. 2024Generative AI for RF Sensing in IoT systemscited by 6
  7. 2024Enhance Modality Robustness in Text-Centric Multimodal Alignment with Adversarial Promptingcited by 5
  8. 2024Enhance the Robustness of Text-Centric Multimodal Alignmentscited by 4
  9. 2025Uncertainty Profiles for LLMs: Uncertainty Source Decomposition and Adaptive Model-Metric Selectioncited by 4
  10. 2024Benchmarking Large Language Model Uncertainty for Prompt Optimizationcited by 2
  11. 2025Multimodal Chip Physical Design Engineer Assistantcited by 1
  12. 2026Trustworthy Agent Network: Trust in Agent Networks Must Be Baked In, Not Bolted Oncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models
  2. 2026Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs
  3. 2023How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model
  4. 2025Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
  5. 2026Multimodal LLMs under Pairwise Modalities
  6. 2026TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs
  7. 2026Anisotropic Modality Align
  8. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques