# OnPrem.LLM: A Privacy-Conscious Document Intelligence Toolkit

arXiv: https://arxiv.org/abs/2505.07672 · HTML: https://mlsanity.com/paper/2505.07672/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 13 references that are in the map.
- First posted on arXiv 12 May 2025.

## Builds on (most cited first, 6 of 6)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Efficient Few-Shot Learning Without Prompts](https://mlsanity.com/paper/2209.11055/) (2022)
- [GPT4All: An Ecosystem of Open Source Compressed Language Models](https://mlsanity.com/paper/2311.04931/) (2023)

## Built on by (most cited first, 2 of 2)

- [Generative AI for FFRDCs](https://mlsanity.com/paper/2509.21040/) (2025)
- [Reflective Personalization Optimization: A Post-hoc Rewriting Framework for Black-Box Large Language Models](https://mlsanity.com/paper/2511.05286/) (2025)

## Abstract (arXiv)

We present OnPrem.LLM, a Python-based toolkit for applying large language models (LLMs) to sensitive, non-public data in offline or restricted environments. The system is designed for privacy-preserving use cases and provides prebuilt pipelines for document processing and storage, retrieval-augmented generation (RAG), information extraction, summarization, classification, and prompt/output processing with minimal configuration. OnPrem.LLM supports multiple LLM backends -- including llama.cpp, Ollama, vLLM, and Hugging Face Transformers -- with quantized model support, GPU acceleration, and seamless backend switching. Although designed for fully local execution, OnPrem.LLM also supports integration with a wide range of cloud LLM providers when permitted, enabling hybrid deployments that balance performance with data control. A no-code web interface extends accessibility to non-technical users.