# Solving AI Foundational Model Latency with Telco Infrastructure

arXiv: https://arxiv.org/abs/2504.03708 · HTML: https://mlsanity.com/paper/2504.03708/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Mar 2025.

## Built on by (most cited first, 3 of 3)

- [Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Survey](https://mlsanity.com/paper/2601.03181/) (2026)
- [Transforming User Defined Criteria into Explainable Indicators with an Integrated LLM AHP System](https://mlsanity.com/paper/2601.05267/) (2025)
- [A Communication-Centric 6G-LLM Architecture for Scalable Tactical Autonomous Defense Vehicle Networks](https://mlsanity.com/paper/2606.01312/) (2026)

## Abstract (arXiv)

Latency remains a critical bottleneck for deploying foundational artificial intelligence (AI) models, such as large language models (LLMs), in customer-facing, real-time applications. While cloud-based inference offers scalability, it frequently introduces delays unacceptable for interactive experiences, such as semantic search, personalized recommendations, or conversational interfaces. Telecommunications operators, historically adept at solving content latency challenges through partnerships with providers like Google and Facebook, now have a unique opportunity to address similar AI latency concerns. This paper presents a technical framework leveraging Telco infrastructure-spanning regional data centers, existing content delivery network (CDN) nodes, and near-radio access network (RAN) sites-as hierarchical "AI edges" for caching and partial inference. We explore the architectural feasibility of embedding semantic and vector-based AI inference caches within existing Telco assets, proposing tiered caching strategies and split-inference architectures that significantly reduce latency and compute costs. Additionally, we address technical challenges specific to Telcos, such as cache synchronization, model distribution, privacy, and hardware acceleration considerations. Finally, we discuss viable partnership models between telcos and AI providers, highlighting how this innovative use of telco infrastructure can unlock both improved AI user experience and new revenue streams.