An Empirical Study of Observability Limits in Advanced Software Supply Chain Attacks

Advanced software supply chain (SSC) attacks are increasingly runtime-only and leave fragmented evidence across hosts, services, and build/dependency layers, making any single telemetry stream insufficient for chain reconstruction. Despite this, no existing dataset provides multi-source runtime monitoring data with end-to-end chain-level ground truth for SSC attacks, leaving the observability limits of such attacks poorly understood. We present SynthChain, a multi-source runtime dataset with chain-level ground truth derived from real-world malicious packages and exploit campaigns, and use it to empirically study observability limits in SSC attacks. SynthChain covers seven representative SSC exploit scenarios across PyPI, npm, and C++ supply chains, spanning Windows, Linux, and containerized environments, with annotations for 14 MITRE ATT&CK tactics, 161 techniques, and 2,919 manually verified Indicators of Compromise (IOC) annotations across 22 log files spanning 11 telemetry types. Our observability analysis shows that no single source is chain-complete: even the best single source recovers fewer than 40% of expected attack steps. Fusing just two complementary sources improves reconstruction by roughly (1.6×), but gains depend on which sources are combined rather than how many. We identify three systematic failure modes---missing-phase gaps, attribution breaks, and ambiguity---and derive telemetry planning guidelines that do not require prior knowledge of specific attacks. A preliminary sensitivity analysis confirms that the multi-source advantage persists under reduced per-source sampling rates. The corpus (approximately 0.59 million events) is released with ground truth and artifacts to support reproducible evaluation of runtime SSC defenses.