# BeLLMan: Controlling LLM Congestion

arXiv: https://arxiv.org/abs/2510.15330 · HTML: https://mlsanity.com/paper/2510.15330/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 57 references that are in the map.
- First posted on arXiv 17 Oct 2025.

## Builds on (most cited first, 16 of 16)

- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [jina-embeddings-v3: Multilingual Embeddings With Task LoRA](https://mlsanity.com/paper/2409.10173/) (2024)
- [DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency](https://mlsanity.com/paper/2408.00741/) (2024)
- [Non-Determinism of "Deterministic" LLM Settings](https://mlsanity.com/paper/2408.04667/) (2024)
- [S3: Increasing GPU Utilization during Generative Inference for Higher Throughput](https://mlsanity.com/paper/2306.06000/) (2023)
- [Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates](https://mlsanity.com/paper/2408.13006/) (2024)
- [HexGen: Generative Inference of Large Language Model over Heterogeneous Environment](https://mlsanity.com/paper/2311.11514/) (2023)
- [Prompt-Based Length Controlled Generation with Multiple Control Types](https://mlsanity.com/paper/2406.10278/) (2024)
- [Aladdin: Joint Placement and Scaling for SLO-Aware LLM Serving](https://mlsanity.com/paper/2405.06856/) (2024)
- [ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor](https://mlsanity.com/paper/2505.09142/) (2025)
- [Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancing](https://mlsanity.com/paper/2408.13510/) (2024)

## Abstract (arXiv)

Large language model (LLM) applications are blindfolded to the infrastructure underneath and generate tokens autoregressively, indifferent to the system load, thus risking inferencing latency inflation and poor user experience. Our first-cut controller, named beLLMan, enables the LLM infrastructure to actively and progressively signal the first-party LLM application to adjust the output length in response to changing system load. On a real testbed with H100 GPUs, beLLMan helps keep inferencing latency under control (upto 8X lower end-to-end latency) and reduces energy consumption by 25% (while serving 19% more requests) during periods of congestion for a summarization workload.