mlsanityData to 14 Oct 2026

Context-Aware Language Models for Forecasting Market Impact from Sequences of Financial News

arXiv 2509.12519 ↗First posted 15 Sep 2025cs.CE cs.CL q-fin.CPcited by 0

Not on the map. Why?

Abstract (arXiv)

Financial news plays a critical role in the information diffusion process in financial markets and is a known driver of stock prices. However, the information in each news article is not necessarily self-contained, often requiring a broader understanding of the historical news coverage for accurate interpretation. Further, identifying and incorporating the most relevant contextual information presents significant challenges. In this work, we explore the value of historical context in the ability of large language models to understand the market impact of financial news. We find that historical context provides a consistent and significant improvement in performance across methods and time horizons. To this end, we propose an efficient and effective contextualization method that uses a large LM to process the main article, while a small LM encodes the historical context into concise summary embeddings that are then aligned with the large model's representation space. We explore the behavior of the model through multiple qualitative and quantitative interpretability tests and reveal insights into the value of contextualization. Finally, we demonstrate that the value of historical context in model predictions has real-world applications, translating to substantial improvements in simulated investment performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 67 references that are in the map. First posted on arXiv 15 Sep 2025.

Builds on

36 of 67 references.

References · 36

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  12. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
24 more
  1. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  2. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  3. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  5. 2022A Survey on In-context Learningcited by 721
  6. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 385
  7. 2023Adapting Language Models to Compress Contextscited by 356
  8. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
  9. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  10. 2023PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Financecited by 222
  11. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 192
  12. 2023TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Seriescited by 169
  13. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 132
  14. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 128
  15. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 121
  16. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  17. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  18. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  19. 2017Listening to Chaotic Whispers: A Deep Learning Framework for News-oriented Stock Trend Predictioncited by 60
  20. 2020Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matchingcited by 35
  21. 2023Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answeringcited by 28
  22. 2022Multi-CLS BERT: An Efficient Alternative to Traditional Ensemblingcited by 14
  23. 2024XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inferencecited by 12
  24. 2022Benchmarking Machine Learning Models to Predict Corporate Bankruptcycited by 4

Built on by

Not cited yet.