# Improving LLM Performance Through Black-Box Online Tuning: A Case for Adding System Specs to Factsheets for Trusted AI

arXiv: https://arxiv.org/abs/2603.11340 · HTML: https://mlsanity.com/paper/2603.11340/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 27 references that are in the map.
- First posted on arXiv 11 Mar 2026.

## Builds on (most cited first, 10 of 10)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding](https://mlsanity.com/paper/2402.05109/) (2024)
- [The relationship between trust in AI and trustworthy machine learning technologies](https://mlsanity.com/paper/1912.00782/) (2019)
- [Socially Responsible AI Algorithms: Issues, Purposes, and Challenges](https://mlsanity.com/paper/2101.02032/) (2021)
- [Challenges of Human-Aware AI Systems](https://mlsanity.com/paper/1910.07089/) (2019)
- [SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines](https://mlsanity.com/paper/2408.04323/) (2024)
- [PixLift: Accelerating Web Browsing via AI Upscaling](https://mlsanity.com/paper/2502.08995/) (2025)

## Abstract (arXiv)

In this paper, we present a novel black-box online controller that uses only end-to-end measurements over short segments, without internal instrumentation, and hill climbing to maximize goodput, defined as the throughput of requests that satisfy the service-level objective. We provide empirical evidence that this design is well-founded. Using this advance in LLM serving as a concrete example, we then discuss the importance of integrating system performance and sustainability metrics into Factsheets for organizations adopting AI systems.