mlsanityData to 14 Oct 2026

TI-JEPA: An Innovative Energy-based Joint Embedding Strategy for Text-Image Multimodal Systems

arXiv 2503.06380 ↗First posted 9 Mar 2025cs.CV cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

This paper focuses on multimodal alignment within the realm of Artificial Intelligence, particularly in text and image modalities. The semantic gap between the textual and visual modality poses a discrepancy problem towards the effectiveness of multi-modalities fusion. Therefore, we introduce Text-Image Joint Embedding Predictive Architecture (TI-JEPA), an innovative pre-training strategy that leverages energy-based model (EBM) framework to capture complex cross-modal relationships. TI-JEPA combines the flexibility of EBM in self-supervised learning to facilitate the compatibility between textual and visual elements. Through extensive experiments across multiple benchmarks, we demonstrate that TI-JEPA achieves state-of-the-art performance on multimodal sentiment analysis task (and potentially on a wide range of multimodal-based tasks, such as Visual Question Answering), outperforming existing pre-training methodologies. Our findings highlight the potential of using energy-based framework in advancing multimodal fusion and suggest significant improvements for downstream applications.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 25 of its 39 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

25 of 39 references.

References · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  6. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  7. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 270
  8. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 249
  9. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  10. 2021Contrastive Learning for Neural Topic Modelcited by 48
  11. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  12. 2024Modeling Dynamic Topics in Chain-Free Fashion by Evolution-Tracking Contrastive Learning and Unassociated Word Exclusioncited by 16
13 more
  1. 2022Adaptive Contrastive Learning on Multimodal Transformer for Review Helpfulness Predictionscited by 13
  2. 2024KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learningcited by 13
  3. 2023Improving Multimodal Sentiment Analysis: Supervised Angular Margin-based Contrastive Learning for Enhanced Fusion Representationcited by 12
  4. 2024Topic Modeling as Multi-Objective Contrastive Optimizationcited by 12
  5. 2023DemaFormer: Damped Exponential Moving Average Transformer with Energy-Based Modeling for Temporal Language Groundingcited by 11
  6. 2024Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generationcited by 9
  7. 2024Encoding and Controlling Global Semantics for Long-form Video Question Answeringcited by 8
  8. 2024MAMA: Meta-optimized Angular Margin Contrastive Framework for Video-Language Representation Learningcited by 8
  9. 2023Expand BERT Representation with Visual Information via Grounded Language Learning with Multimodal Partial Alignmentcited by 5
  10. 2024Energy-Based Models with Applications to Speech and Language Processingcited by 5
  11. 2024Multi-Scale Contrastive Learning for Video Temporal Groundingcited by 5
  12. 2025Enhancing Multimodal Entity Linking with Jaccard Distance-based Conditional Contrastive Learning and Contextual Visual Augmentationcited by 3
  13. 2022Vision-and-Language Pretrainingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learningcited by 17
  2. 2026Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddingscited by 5
  3. 2025CrossJEPA: Cross-Modal Joint-Embedding Predictive Architecture for Efficient 3D Representation Learning from 2D Imagescited by 4
  4. 2025Time to Embed: Unlocking Foundation Models for Time Series with Channel Descriptionscited by 2
  5. 2026The JEPA Paradox in Language: The Geometry of Linguistic Alternativescited by 2
  6. 2026AlertBERT: A noise-robust alert grouping framework for simultaneous cyber attackscited by 0
  7. 2026The JEPA Predictor: A Transferable Operator for Occluded Feature Completioncited by 0
  8. 2026Bar-JEPA: Extracting Values from Bar Chart with Joint-Embedding Predictive Architecturecited by 0