mlsanityData to 14 Oct 2026

AC-LoRA: (Almost) Training-Free Access Control-Aware Multi-Modal LLMs

arXiv 2505.11557 ↗First posted 15 May 2025cs.CR cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Corporate LLMs are gaining traction for efficient knowledge dissemination and management within organizations. However, as current LLMs are vulnerable to leaking sensitive information, it has proven difficult to apply them in settings where strict access control is necessary. To this end, we design AC-LoRA, an end-to-end system for access control-aware corporate LLM chatbots that maintains a strong information isolation guarantee. AC-LoRA maintains separate LoRA adapters for permissioned datasets, along with the document embedding they are finetuned on. AC-LoRA retrieves a precise set of LoRA adapters based on the similarity score with the user query and their permission. This similarity score is later used to merge the responses if more than one LoRA is retrieved, without requiring any additional training for LoRA routing. We provide an end-to-end prototype of AC-LoRA, evaluate it on two datasets, and show that AC-LoRA matches or even exceeds the performance of state-of-the-art LoRA mixing techniques while providing strong isolation guarantees. Furthermore, we show that AC-LoRA design can be directly applied to different modalities.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 44 of its 62 references that are in the map. First posted on arXiv 15 May 2025.

Builds on

44 of 62 references.

References · 44

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  4. 2020Extracting Training Data from Large Language Modelscited by 2,189
  5. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  6. 2019Machine Unlearningcited by 1,061
  7. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  8. 2023Extracting Training Data from Diffusion Modelscited by 732
  9. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  10. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  11. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  12. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 257
32 more
  1. 2024Mixture of LoRA Expertscited by 181
  2. 2018SpotTune: Transfer Learning through Adaptive Fine-tuningcited by 168
  3. 2023From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacycited by 143
  4. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 127
  5. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 126
  6. 2024Jasper and Stella: distillation of SOTA embedding modelscited by 112
  7. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 94
  8. 2023Soft Merging of Experts with Adaptive Routingcited by 90
  9. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 86
  10. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  11. 2023Bag of Tricks for Training Data Extraction from Language Modelscited by 70
  12. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
  13. 2024Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoningcited by 64
  14. 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 59
  15. 2024Injecting New Knowledge into Large Language Models via Supervised Fine-Tuningcited by 58
  16. 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Modelscited by 56
  17. 2025Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Surveycited by 51
  18. 2024Learn What You Want to Unlearn: Unlearning Inversion Attacks against Machine Unlearningcited by 48
  19. 2025Poison-RAG: Adversarial Data Poisoning Attacks on Retrieval-Augmented Generation in Recommender Systemscited by 32
  20. 2024RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Contentcited by 26
  21. 2025MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attackscited by 24
  22. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 24
  23. 2024MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuningcited by 19
  24. 2024MeteoRA: Multiple-tasks Embedded LoRA for Large Language Modelscited by 16
  25. 2024Memorization in deep learning: A surveycited by 15
  26. 2024MoLE: Enhancing Human-centric Text-to-image Diffusion via Mixture of Low-rank Expertscited by 15
  27. 2024Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learningcited by 14
  28. 2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learningcited by 13
  29. 2025DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanismcited by 11
  30. 2025Privacy-Aware RAG: Secure and Isolated Knowledge Retrievalcited by 10
  31. 2024HDMoLE: Mixture of LoRA Experts with Hierarchical Routing and Dynamic Thresholds for Fine-Tuning LLM-based ASR Modelscited by 9
  32. 2024GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Parametric Memory Decoding for Zero-Shot Routing in LoRA-Based External Parametric Memorycited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Privacy Control in Conversational LLM Platforms: A Walkthrough Study
  2. 2025Permissioned LLMs: Enforcing Access Control in Large Language Models
  3. 2026Harness-MU: A Safe, Governed, and Effective Harness for Multi-User LLM Agents
  4. 2026Auditing Training Data in Domain-adapted LLMs: LoRA-MINT
  5. 2026Securing LLM-as-a-Service for Small Businesses: An Industry Case Study of a Distributed Chatbot Deployment Platform
  6. 2025Unlearning Imperative: Securing Trustworthy and Responsible LLMs through Engineered Forgetting
  7. 2024IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems
  8. 2025Beyond Chat: a Framework for LLMs as Human-Centered Support Systems