mlsanityData to 14 Oct 2026

CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answering

arXiv 2507.22937 ↗First posted 25 Jul 2025cs.CL cs.AIcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

With the rapid evolution of artificial intelligence, AIOps has emerged as a prominent paradigm in DevOps. Lots of work has been proposed to improve the performance of different AIOps phases. However, constrained by domain-specific knowledge, a single model can only handle the operation requirement of a specific task,such as log parser,root cause analysis. Meanwhile, combining multiple models can achieve more efficient results, which have been proved in both previous ensemble learning and the recent LLM training domain. Inspired by these works,to address the similar challenges in AIOPS, this paper first proposes a collaboration-of-expert framework(CoE-Ops) incorporating a general-purpose large language model task classifier. A retrieval-augmented generation mechanism is introduced to improve the framework's capability in handling both Question-Answering tasks with high-level(Code,build,Test,etc.) and low-level(fault analysis,anomaly detection,etc.). Finally, the proposed method is implemented in the AIOps domain, and extensive experiments are conducted on the DevOps-EVAL dataset. Experimental results demonstrate that CoE-Ops achieves a 72% improvement in routing accuracy for high-level AIOps tasks compared to existing CoE methods, delivers up to 8% accuracy enhancement over single AIOps models in DevOps problem resolution, and outperforms larger-scale Mixture-of-Experts (MoE) models by up to 14% in accuracy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 27 of its 50 references that are in the map. First posted on arXiv 25 Jul 2025.

Builds on

27 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May
  2. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM12 Mar

Other references · 25

  1. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  2. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  3. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  4. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  5. 2024More Agents Is All You Needcited by 180
  6. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 165
  7. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 145
  8. 2022Machine Learning Operations (MLOps): Overview, Definition, and Architecturecited by 128
  9. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 95
  10. 2025Universal Model Routing for Efficient LLM Inferencecited by 86
  11. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  12. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inferencecited by 60
13 more
  1. 2025MixLLM: Dynamic Routing in Mixed Large Language Modelscited by 59
  2. 2023Getting MoRE out of Mixture of Language Model Reasoning Expertscited by 44
  3. 2024MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMscited by 41
  4. 2024Eagle: Efficient Training-Free Router for Multi-LLM Inferencecited by 35
  5. 2025Leveraging Uncertainty Estimation for Efficient LLM Routingcited by 21
  6. 2019A Survey of DevOps Concepts and Challengescited by 12
  7. 2024SelectLLM: Query-Aware Efficient Selection Algorithm for Large Language Modelscited by 8
  8. 2024Bench-CoE: a Framework for Collaboration of Experts from Benchmarkcited by 7
  9. 2021Collaboration of Experts: Achieving 80% Top-1 Accuracy on ImageNet with 100M FLOPscited by 5
  10. 2024Composition of Experts: A Modular Compound AI System Leveraging Large Language Modelscited by 5
  11. 2025CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memorycited by 5
  12. 2025LADs: Leveraging LLMs for AI-Driven DevOpscited by 4
  13. 2024Towards MLOps: A DevOps Tools Recommender System for Machine Learning Systemcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating LLM Reasoning in the Operations Research Domain with ORQA
  2. 2026OpsLLM: Construction of Large Language Model for Software Operations with Multi-stage Learning
  3. 2025MSCoRe: A Benchmark for Multi-Stage Collaborative Reasoning in LLM Agents
  4. 2023AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs
  5. 2026Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety
  6. 2025Towards Conversational AI for Human-Machine Collaborative MLOps
  7. 2024Creating an LLM-based AI-agent: A high-level methodology towards enhancing LLMs with APIs
  8. 2026SciOrch: Learning to Orchestrate Expert LLMs for Solving Frontier Multimodal Scientific Reasoning Tasks