What Does an Observability Foundation Model Know?
New in Mechanistic interpretability; no direction yet. Mechanistic interpretability
Abstract (arXiv)
A linear probe can show that a label is recoverable from a model's hidden states, but not whether that goes beyond what the input already reveals, or whether the model uses it. We audit Toto, an observability forecasting foundation model, on the Benchmark of Observability Metrics (BOOM) across five series-disjoint resplits, comparing linear probes on its frozen residual stream with models that read the raw input window and with Toto's architecture stripped of its trained configuration. Short-vs-medium cadence and metric type are more linearly recoverable from Toto's residuals than from the strongest raw-window model in every resplit (macro-F1 0.766 vs. 0.633 and 0.545 vs. 0.498). Domain is nearly tied, and series cardinality is recovered far better from the raw window. MOMENT-base shows related cadence, metric-type, and domain readouts. Recoverability is not use: exchanging Toto's residuals with those of high-burst donors moves a future-burstiness readout as intended but does not make forecasts consistently burstier than a randomized donor. A BOOM-trained coordination probe has negative zero-shot R^2 on the tested external benchmarks. We report each label against its strongest baseline.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 31 references that are in the map. First posted on arXiv 4 Oct 2026.
Builds on
23 of 31 references.
References · 23
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
- 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
- 2016Understanding intermediate layers using linear classifier probescited by 1,353
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
- 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
- 2020A Primer in BERTology: What we know about how BERT workscited by 815
- 2024Chronos: Learning the Language of Time Seriescited by 759
- 2023A decoder-only foundation model for time-series forecastingcited by 692
- 2019Designing and Interpreting Probes with Control Taskscited by 640
11 more
- 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 482
- 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
- 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
- 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
- 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 205
- 2020Forecasting day-ahead electricity prices: A review of state-of-the-art algorithms, best practices and an open-access benchmarkcited by 108
- 2025This Time is Different: An Observability Perspective on Time Series Foundation Modelscited by 68
- 2025fev-bench: A Realistic Benchmark for Time Series Forecastingcited by 64
- 2024Toto: Time Series Optimized Transformer for Observabilitycited by 48
- 2024Exploring Representations and Interventions in Time Series Foundation Modelscited by 20
- 2025On the Internal Semantics of Time-Series Foundation Modelscited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026When, Not How Much: Evaluating Time-Series Foundation Models on Sparse Events
- 2026GRAM: Correcting Frozen Time-Series Foundation Models via Graph-Retrieved Amplitude Memory
- 2026Correct then Forecast: Observer State-Space Models for Time Series Forecasting
- 2026AI Observability for Large Language Model Systems: A Multi-Layer Analysis of Monitoring Approaches from Confidence Calibration to Infrastructure Tracing ↗
- 2026Foundations without Fundamentals: Zero-Shot Blind Spots in Time Series FMs
- 2026Universal Redundancies in Time Series Foundation Models
- 2025Rethinking Evaluation in the Era of Time Series Foundation Models: (Un)known Information Leakage Challenges
- 2026Prediction Bottlenecks Don't Discover Causal Structure (But Here's What They Actually Do)