mlsanityData to 14 Oct 2026

Enabling Performant and Flexible Model-Internal Observability for LLM Inference

arXiv 2605.11093 ↗First posted 11 May 2026cs.LG cs.AI cs.PFcited by 0

Not on the map. Why?

Abstract (arXiv)

Today's inference-time workloads increasingly depend on timely access to a model's internal states. We present DMI-Lib, a high-speed deep model inspector that treats internal observability as a first-class systems primitive, decoupling it from the inference hot path via an asynchronous observability substrate built from Ring^2, a GPU-CPU memory abstraction for capturing and staging tensors, and a policy-controlled host backend that exports them. DMI-Lib enables the placement of observation points across a rich space of internal signals and diverse inference backends while preserving serving optimizations and adhering to tight GPU memory budgets. Our experiments demonstrate that DMI-Lib incurs only 0.4%--6.8% overhead in offline batch inference and an average of 6% in moderate online serving, reducing latency overhead by 2x-15x compared to existing baselines with similar observability features. DMI-Lib is open-sourced at https://github.com/ProjectDMX/DMI.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 51 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

37 of 51 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2024Mixtral of Expertscited by 1,823
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
25 more
  1. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  2. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  4. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  5. 2023Steering Language Models With Activation Engineeringcited by 922
  6. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  8. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 547
  9. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  10. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  11. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 160
  12. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 118
  13. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  14. 2025Detecting High-Stakes Interactions with Activation Probescited by 53
  15. 2026Building Production-Ready Probes For Geminicited by 47
  16. 2025ICR Probe: Tracking Hidden State Dynamics for Reliable Hallucination Detection in LLMscited by 32
  17. 2026DART: Diffusion-Inspired Speculative Decoding for Fast LLM Inferencecited by 24
  18. 2025Advancing LLM Safe Alignment with Safety Representation Rankingcited by 16
  19. 2024Minder: Faulty Machine Detection for Large-scale Distributed Model Trainingcited by 11
  20. 2025Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Trainingcited by 6
  21. 2025Reliable and Resilient Collective Communication Library for LLM Training and Servingcited by 4
  22. 2026Make Every Draft Count: Hidden State based Speculative Decodingcited by 4
  23. 2026vLLM Hook v0: A Plug-in for Programming Model Internals on vLLMcited by 4
  24. 2025Precise Attribute Intensity Control in Large Language Models via Targeted Representation Editingcited by 2
  25. 2023FlexModel: A Framework for Interpretability of Distributed Large Language Modelscited by 1

Built on by

Not cited yet.