# CLIPTime: Time-Aware Multimodal Representation Learning from Images and Text

arXiv: https://arxiv.org/abs/2508.00447 · HTML: https://mlsanity.com/paper/2508.00447/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agricultural vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 1 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Agricultural vision-language models](https://mlsanity.com/mllm/agricultural-vision-language-models/)

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [CogVLM2: Visual Language Models for Image and Video Understanding](https://mlsanity.com/paper/2408.16500/) (2024)
- [Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning](https://mlsanity.com/paper/2308.06038/) (2023)
- [C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion](https://mlsanity.com/paper/2403.14119/) (2024)
- [Time-VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting](https://mlsanity.com/paper/2502.04395/) (2025)
- [VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought](https://mlsanity.com/paper/2406.14596/) (2024)
- [Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions](https://mlsanity.com/paper/2501.04437/) (2025)
- [Rethinking VLMs and LLMs for Image Classification](https://mlsanity.com/paper/2410.14690/) (2024)
- [Leveraging Temporal Contextualization for Video Action Recognition](https://mlsanity.com/paper/2404.09490/) (2024)
- [From Images to Signals: Are Large Vision Models Useful for Time Series Analysis?](https://mlsanity.com/paper/2505.24030/) (2025)
- [Regression in EO: Are VLMs Up to the Challenge?](https://mlsanity.com/paper/2502.14088/) (2025)
- [PV-VLM: A Multimodal Vision-Language Approach Incorporating Sky Images for Intra-Hour Photovoltaic Power Forecasting](https://mlsanity.com/paper/2504.13624/) (2025)
- [Synthetic Fungi Datasets: A Time-Aligned Approach](https://mlsanity.com/paper/2501.02855/) (2025)
- [FungalZSL: Zero-Shot Fungal Classification with Image Captioning Using a Synthetic Data Approach](https://mlsanity.com/paper/2502.19038/) (2025)

## Abstract (arXiv)

Understanding the temporal dynamics of biological growth is critical across diverse fields such as microbiology, agriculture, and biodegradation research. Although vision-language models like Contrastive Language Image Pretraining (CLIP) have shown strong capabilities in joint visual-textual reasoning, their effectiveness in capturing temporal progression remains limited. To address this, we propose CLIPTime, a multimodal, multitask framework designed to predict both the developmental stage and the corresponding timestamp of fungal growth from image and text inputs. Built upon the CLIP architecture, our model learns joint visual-textual embeddings and enables time-aware inference without requiring explicit temporal input during testing. To facilitate training and evaluation, we introduce a synthetic fungal growth dataset annotated with aligned timestamps and categorical stage labels. CLIPTime jointly performs classification and regression, predicting discrete growth stages alongside continuous timestamps. We also propose custom evaluation metrics, including temporal accuracy and regression error, to assess the precision of time-aware predictions. Experimental results demonstrate that CLIPTime effectively models biological progression and produces interpretable, temporally grounded outputs, highlighting the potential of vision-language models in real-world biological monitoring applications.