# MemGuard-Alpha: Limits of Membership Inference for Detecting and Filtering Memorization-Contaminated Signals in LLM-Based Financial Forecasting

arXiv: https://arxiv.org/abs/2603.26797 · HTML: https://mlsanity.com/paper/2603.26797/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Membership inference attacks, one of 12 directions in LLM safety and security.
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 26 Mar 2026.

## Placed in

- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 14 of 14)

- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [Large Language Models: An Applied Econometric Framework](https://mlsanity.com/paper/2412.07031/) (2024)
- [Assessing Look-Ahead Bias in Stock Return Predictions Generated By GPT Sentiment Analysis](https://mlsanity.com/paper/2309.17322/) (2023)
- [Financial Statement Analysis with Large Language Models](https://mlsanity.com/paper/2407.17866/) (2024)
- [Can LLM-based Financial Investing Strategies Outperform the Market in Long Run?](https://mlsanity.com/paper/2505.07078/) (2025)
- [Your AI, Not Your View: The Bias of LLMs in Investment Analysis](https://mlsanity.com/paper/2507.20957/) (2025)
- [Look-Ahead-Bench: a Standardized Benchmark of Look-ahead Bias in Point-in-Time LLMs for Finance](https://mlsanity.com/paper/2601.13770/) (2026)
- [Evaluating LLMs in Finance Requires Explicit Bias Consideration](https://mlsanity.com/paper/2602.14233/) (2026)
- [A Fast and Effective Solution to the Problem of Look-ahead Bias in LLMs](https://mlsanity.com/paper/2512.06607/) (2025)
- [Detecting Lookahead Bias in LLM Forecasts](https://mlsanity.com/paper/2512.23847/) (2025)
- [Time Machine GPT](https://mlsanity.com/paper/2404.18543/) (2024)
- [Can Blindfolded LLMs Still Trade? An Anonymization-First Framework for Portfolio Optimization](https://mlsanity.com/paper/2603.17692/) (2026)

## Abstract (arXiv)

Large language models are increasingly used to generate financial alpha signals, but many have memorized the historical data in their training corpora, producing apparent accuracy that collapses out of sample. Membership inference attacks (MIA) have been proposed as a diagnostic. What has not been established is whether MIA scores are informative about memorization in this setting, or whether signal-level filtering built on them helps once realistic costs are applied.We introduce MemGuard-Alpha, comprising a composite contamination score combining five MIA methods with a temporal proximity feature, and Cross-Model Memorization Disagreement, which exploits variation in training cutoffs across models. We then audit both. Across seven LLMs (124M-7B), 50 S&P 100 constituents, 42,800 prompts and 299,600 prompt-model MIA scores spanning 2019-2024, three findings emerge.First, where in-sample status is defined by a training cutoff, a temporal proximity feature recovers that label perfectly (ROC-AUC 1.000) because it is a monotone transform of the defining variable; any composite score containing such a feature reports separation that is arithmetic rather than detection. Second, the discriminative power of the MIA scores is largely attributable to scale differences between models: asked at a fixed date which models had that date in training, raw scores appear highly informative (AUC up to 0.99), but under three independent within-model normalizations discrimination falls to 0.487-0.537. Third, with transaction costs applied symmetrically, no filtering variant improves risk-adjusted performance over the unfiltered ensemble, none attains significant Fama-French five-factor alpha, and excluding the single weakest model outperforms every contamination-based filter.We report these as negative results with the failure modes that produced them, and release all artifacts needed to reproduce them.