mlsanityData to 14 Oct 2026

Temporal Leakage in Financial News NLP: A Multi-Architecture Audit with a Regime-Specific M&A Signal

arXiv 2608.17223 ↗First posted 18 Aug 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Financial-news direction prediction has become a popular NLP benchmark, yet reported gains depend critically on whether the train-test split is chronological or random, i.e., on temporal leakage. We audit this dependence on a 49,799-article corpus across 16 feature-model combinations spanning TF-IDF, MiniLM, FinBERT, and fine-tuned RoBERTa-large / DeBERTa-v3-large, plus separate zero/few-shot and LoRA probes of Llama-3 and Qwen2.5 LLMs: random splits inflate MCC by 1.1× to 6.5×, tracking model capacity and feature richness, and end-to-end FinBERT fine-tuning re-amplifies rather than closes the gap (size-matched ratio 1.75×). Conditioning on event type, mergers and acquisitions (M&A) is the only audited category with a positive locked-test signal under near-temporal chronological evaluation (TF-IDF MCC = 0.138 train-only, 0.068 under traincupval refit; 10,000-permutation p < 10-3); the signal does not transfer to FNSPID's 2009-2020 U.S. corpus, localising the headline to our 2024-2025 European-tilted M&A semantics rather than a universal predictor. Three independent role labellers converge on acquirer-tagged articles as the signal locus, a power-limited qualitative convergence rather than a hypothesis-tested asymmetry. Chronological splitting plays for financial NLP the role characteristics-purging plays for asset pricing: it strips the predictable, stale component of news and leaves a residual that is small, event-localized, and lexically shallow. We advocate leakage audits as a required disclosure for financial-NLP benchmarks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 56 references that are in the map. First posted on arXiv 18 Aug 2026.

Builds on

16 of 56 references.

References · 16

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  3. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  5. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  6. 2023BloombergGPT: A Large Language Model for Financecited by 939
  7. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 385
  8. 2022Data Contamination: From Memorization to Exploitationcited by 236
  9. 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
  10. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 178
  11. 2020FinBERT: A Pretrained Language Model for Financial Communicationscited by 161
  12. 2020With Little Power Comes Great Responsibilitycited by 142
4 more
  1. 2024FinBen: A Holistic Financial Benchmark for Large Language Modelscited by 141
  2. 2020We Need to Talk About Random Splitscited by 71
  3. 2017Listening to Chaotic Whispers: A Deep Learning Framework for News-oriented Stock Trend Predictioncited by 60
  4. 2024LLMFactor: Extracting Profitable Factors through Prompts for Explainable Stock Movement Predictioncited by 33

Built on by

Not cited yet.