# Embedded AI Companion System on Edge Devices

arXiv: https://arxiv.org/abs/2601.08128 · HTML: https://mlsanity.com/paper/2601.08128/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 26 references that are in the map.
- First posted on arXiv 13 Jan 2026.

## Builds on (most cited first, 9 of 9)

- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Hello Again! LLM-powered Personalized Agent for Long-term Dialogue](https://mlsanity.com/paper/2406.05925/) (2024)
- [Towards Lifelong Dialogue Agents via Timeline-based Memory Management](https://mlsanity.com/paper/2406.10996/) (2024)
- [SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMs](https://mlsanity.com/paper/2505.17052/) (2025)
- [AutoPal: Autonomous Adaptation to Users for Personal AI Companionship](https://mlsanity.com/paper/2406.13960/) (2024)
- [Towards Adaptive Context Management for Intelligent Conversational Question Answering](https://mlsanity.com/paper/2509.17829/) (2025)

## Abstract (arXiv)

Computational resource constraints on edge devices make it difficult to develop a fully embedded AI companion system with a satisfactory user experience. AI companion and memory systems detailed in existing literature cannot be directly used in such an environment due to lack of compute resources and latency concerns. In this paper, we propose a memory paradigm that alternates between active and inactive phases: during phases of user activity, the system performs low-latency, real-time dialog using lightweight retrieval over existing memories and context; whereas during phases of user inactivity, it conducts more computationally intensive extraction, consolidation, and maintenance of memories across full conversation sessions. This design minimizes latency while maintaining long-term personalization under the tight constraints of embedded hardware. We also introduce an AI Companion benchmark designed to holistically evaluate the AI Companion across both its conversational quality and memory capabilities. In our experiments, we found that our system (using a very weak model: Qwen2.5-7B-Instruct quantized int4) outperforms the equivalent raw LLM without memory across most metrics, and performs comparably to GPT-3.5 with 16k context window.