mlsanityData to 14 Oct 2026

Compliance Brain Assistant: Conversational Agentic AI for Assisting Compliance Tasks in Enterprise Environments

arXiv 2507.17289 ↗First posted 23 Jul 2025cs.AIcited by 4

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

This paper presents Compliance Brain Assistant (CBA), a conversational, agentic AI assistant designed to boost the efficiency of daily compliance tasks for personnel in enterprise environments. To strike a good balance between response quality and latency, we design a user query router that can intelligently choose between (i) FastTrack mode: to handle simple requests that only need additional relevant context retrieved from knowledge corpora; and (ii) FullAgentic mode: to handle complicated requests that need composite actions and tool invocations to proactively discover context across various compliance artifacts, and/or involving other APIs/models for accommodating requests. A typical example would be to start with a user query, use its description to find a specific entity and then use the entity's information to query other APIs for curating and enriching the final AI response. Our experimental evaluations compared CBA against an out-of-the-box LLM on various real-world privacy/compliance-related queries targeting various personas. We found that CBA substantially improved upon the vanilla LLM's performance on metrics such as average keyword match rate (83.7% vs. 41.7%) and LLM-judge pass rate (82.0% vs. 20.0%). We also compared metrics for the full routing-based design against the `fast-track only` and `full-agentic` modes and found that it had a better average match-rate and pass-rate while keeping the run-time approximately the same. This finding validated our hypothesis that the routing mechanism leads to a good trade-off between the two worlds.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 7 of its 13 references that are in the map. First posted on arXiv 23 Jul 2025.

Builds on

7 of 13 references.

References · 7

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  4. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  5. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  6. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  7. 2024Ingest-And-Ground: Dispelling Hallucinations from Continually-Pretrained LLMs with RAGcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Your Agents Are Aging Too: Agent Lifespan Engineering for Deployed Systemscited by 9
  2. 2026LLM Agents in Law: Taxonomy, Applications, and Challengescited by 5
  3. 2026Agent Benchmarks Fail Public Sector Requirementscited by 3
  4. 2026GMPilot: An Expert AI Agent For FDA cGMP Compliancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024AuditNet: A Conversational AI-based Security Assistant [DEMO]
  2. 2026PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?
  3. 2025Overhearing LLM Agents: A Survey, Taxonomy, and Roadmap
  4. 2025HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
  5. 2026Overseeing Agents Without Constant Oversight: Challenges and Opportunities
  6. 2026From Transcripts to AI Agents: Knowledge Extraction, RAG Integration, and Robust Evaluation of Conversational AI Assistants
  7. 2026NeuroFilter: Activation-Based Guardrails for Privacy-Conscious LLM Agents
  8. 2025AI Chatbots as Professional Service Agents: Developing a Professional Identity