mlsanityData to 14 Oct 2026

TempoFormer: A Transformer for Temporally-aware Representations in Change Detection

arXiv 2408.15689 ↗First posted 28 Aug 2024cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Dynamic representation learning plays a pivotal role in understanding the evolution of linguistic content over time. On this front both context and time dynamics as well as their interplay are of prime importance. Current approaches model context via pre-trained representations, which are typically temporally agnostic. Previous work on modelling context and temporal dynamics has used recurrent methods, which are slow and prone to overfitting. Here we introduce TempoFormer, the first task-agnostic transformer-based and temporally-aware model for dynamic representation learning. Our approach is jointly trained on inter and intra context dynamics and introduces a novel temporal variation of rotary positional embeddings. The architecture is flexible and can be used as the temporal representation foundation of other models or applied to different transformer-based architectures. We show new SOTA performance on three different real-time change detection tasks.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 37 of its 75 references that are in the map. First posted on arXiv 28 Aug 2024.

Builds on

37 of 75 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  11. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 230
  12. 2022TimeLMs: Diachronic Language Models from Twittercited by 131
25 more
  1. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 121
  2. 2019Hierarchical Transformers for Long Document Classificationcited by 104
  3. 2018RumourEval 2019: Determining Rumour Veracity and Support for Rumourscited by 75
  4. 2022Code Synonyms Do Matter: Multiple Synonyms Matching Network for Automatic ICD Codingcited by 53
  5. 2017Dynamic Word Embeddingscited by 51
  6. 2021Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Mediacited by 51
  7. 2021Time Masking for Temporal Language Modelscited by 46
  8. 2019Trouble on the Horizon: Forecasting the Derailment of Online Conversations as they Developcited by 41
  9. 2021Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Studycited by 39
  10. 2021SpanEmo: Casting Multi-label Emotion Classification as Span-predictioncited by 37
  11. 2021Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modelingcited by 35
  12. 2021Rumor Detection on Twitter with Claim-Guided Hierarchical Graph Attention Networkscited by 29
  13. 2019Outlier Detection for Improved Data Quality and Diversity in Dialog Systemscited by 28
  14. 2019A Transformer-based approach to Irony and Sarcasm detectioncited by 28
  15. 2020Event Detection: Gate Diversity and Syntactic Importance Scoresfor Graph Convolution Neural Networkscited by 23
  16. 2022BiTimeBERT: Extending Pre-Trained Language Representations with Bi-Temporal Informationcited by 17
  17. 2022Identifying Moments of Change from Longitudinal User Textcited by 17
  18. 2021Dynamic Forecasting of Conversation Derailmentcited by 16
  19. 2019Do Sentence Interactions Matter? Leveraging Sentence Level Representations for Fake News Classificationcited by 15
  20. 2019Generic Intent Representation in Web Searchcited by 12
  21. 2020Multi-Task Learning with Auxiliary Speaker Identification for Conversational Emotion Recognitioncited by 12
  22. 2023Recurrent Attention Networks for Long-text Modelingcited by 12
  23. 2023Modelling Temporal Document Sequences for Clinical ICD Codingcited by 7
  24. 2024Generation, Distillation and Evaluation of Motivational Interviewing-Style Reflections with a Foundational Language Modelcited by 5
  25. 2023Sig-Networks Toolkit: Signature Networks for Longitudinal Language Modellingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Dynamic Topic Evolution with Temporal Decay and Attention in Large Language Modelscited by 3
  2. 2025Automated Data Enrichment using Confidence-Aware Fine-Grained Debate among Open-Source LLMs for Mental Health and Online Safetycited by 0
  3. 2026LiFT: How to Enable In-Context Learning for Longitudinal Modellingcited by 0
  4. 2026CUNY at CLPsych 2026: A Pipeline Approach to Classification and Summarization of Mental Health Changescited by 0
  5. 2026psytechlab at CLPsych 2026: Utilising Natural Language Processing methods and Large Language Models for Social Media Text Analysiscited by 0