mlsanityData to 7 Oct 2026

TorchOpera: A Compound AI System for LLM Safety

arXiv 2406.10847 ↗First posted 16 Jun 2024cs.AI cs.CE cs.CLcited by 4

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

We introduce TorchOpera, a compound AI system for enhancing the safety and quality of prompts and responses for Large Language Models. TorchOpera ensures that all user prompts are safe, contextually grounded, and effectively processed, while enhancing LLM responses to be relevant and high quality. TorchOpera utilizes the vector database for contextual grounding, rule-based wrappers for flexible modifications, and specialized mechanisms for detecting and adjusting unsafe or incorrect content. We also provide a view of the compound AI system to reduce the computational cost. Extensive experiments show that TorchOpera ensures the safety, reliability, and applicability of LLMs in real-world settings while maintaining the efficiency of LLM responses.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 45 of its 63 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

45 of 63 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming6 Apr
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2023Mistral 7Bcited by 3,065
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  11. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  12. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
31 more
  1. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  2. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  3. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  4. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  5. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  6. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  7. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  8. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  9. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  10. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  11. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  12. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  13. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  14. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 324
  15. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  16. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  17. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  18. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  19. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  20. 2024LLM Multi-Agent Systems: Challenges and Open Problemscited by 119
  21. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  22. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 87
  23. 2024Stable LM 2 1.6B Technical Reportcited by 86
  24. 2023BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agentscited by 83
  25. 2023Punica: Multi-Tenant LoRA Servingcited by 80
  26. 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 74
  27. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Cluescited by 57
  28. 2023RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Modelscited by 36
  29. 2023Efficient Toxic Content Detection by Bootstrapping and Distilling Large Language Modelscited by 21
  30. 2022CVNets: High Performance Library for Computer Visioncited by 16
  31. 2023On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlookcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2024Robust Planning with Compound LLM Architectures: An LLM-Modulo Approachcited by 8
  2. 2025Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directionscited by 4
  3. 2026Design Methodology and Performance Trade-offs Management for Distributed and Compound AI Systemscited by 1
  4. 2026Atlas: Optimizing Deployment of Compound AI Workflows on Heterogeneous Clusterscited by 0