mlsanityData to 7 Oct 2026

Tele-Knowledge Pre-training for Fault Analysis

arXiv 2210.11298 ↗First posted 20 Oct 2022cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

In this work, we share our experience on tele-knowledge pre-training for fault analysis, a crucial task in telecommunication applications that requires a wide range of knowledge normally found in both machine log data and product documents. To organize this knowledge from experts uniformly, we propose to create a Tele-KG (tele-knowledge graph). Using this valuable data, we further propose a tele-domain language pre-training model TeleBERT and its knowledge-enhanced version, a tele-knowledge re-training model KTeleBERT. which includes effective prompt hints, adaptive numerical data encoding, and two knowledge injection paradigms. Concretely, our proposal includes two stages: first, pre-training TeleBERT on 20 million tele-related corpora, and then re-training it on 1 million causal and machine-related corpora to obtain KTeleBERT. Our evaluation on multiple tasks related to fault analysis in tele-applications, including root-cause analysis, event association prediction, and fault chain tracing, shows that pre-training a language model with tele-domain data is beneficial for downstream tasks. Moreover, the KTeleBERT re-training further improves the performance of task models, highlighting the effectiveness of incorporating diverse tele-knowledge into the model.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 43 of its 54 references that are in the map. First posted on arXiv 20 Oct 2022.

Builds on

43 of 54 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  8. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  9. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  10. 2016Complex Embeddings for Simple Link Predictioncited by 897
  11. 2019RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Spacecited by 893
  12. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
31 more
  1. 2017DeepFM: A Factorization-Machine based Neural Network for CTR Predictioncited by 673
  2. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  3. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  4. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 395
  5. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  6. 2019KG-BERT: BERT for Knowledge Graph Completioncited by 323
  7. 2018AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networkscited by 319
  8. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  9. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  10. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 233
  11. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 230
  12. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  13. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  14. 2021LogBERT: Log Anomaly Detection via BERTcited by 114
  15. 2021TABBIE: Pretrained Representations of Tabular Datacited by 111
  16. 2016Neural Photo Editing with Introspective Adversarial Networkscited by 109
  17. 2019StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understandingcited by 108
  18. 2020TUTA: Tree-based Transformers for Generally Structured Table Pre-trainingcited by 100
  19. 2022TableFormer: Robust Transformer Modeling for Table-Text Encodingcited by 90
  20. 2019Multi-view Knowledge Graph Embedding for Entity Alignmentcited by 74
  21. 2021Log-based Anomaly Detection Without Log Parsingcited by 59
  22. 2022Log-based Anomaly Detection with Deep Learning: How Far Are We?cited by 56
  23. 2022Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Taskscited by 42
  24. 2020An Embedding Learning Framework for Numerical Features in CTR Predictioncited by 37
  25. 2020N-LTP: An Open-source Neural Language Technology Platform for Chinesecited by 29
  26. 2022Deep Learning for Anomaly Detection in Log Data: A Surveycited by 27
  27. 2021Knowledge Perceived Multi-modal Pretraining in E-commercecited by 18
  28. 2022LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injectioncited by 18
  29. 2021Billion-scale Pre-trained E-commerce Product Knowledge Graph Modelcited by 13
  30. 2022NeuralKG: An Open Source Library for Diverse Representation Learning of Knowledge Graphscited by 8
  31. 2021Knowledge Graph Embedding in E-commerce Applications: Attentive Reasoning, Explanations, and Transferable Rulescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 7; 2025: 3; 2026: 2.

18 papers.

Sort:

By area:

  1. 2025Time-MQA: Time Series Multi-Task Question Answering with Context Enhancementcited by 74
  2. 2023Making Large Language Models Perform Better in Knowledge Graph Completioncited by 48
  3. 2023OceanGPT: A Large Language Model for Ocean Science Taskscited by 42
  4. 2024TelecomGPT: A Framework to Build Telecom-Specfic Large Language Modelscited by 38
  5. 2023Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answeringcited by 28
  6. 2023When Do Program-of-Thoughts Work for Reasoning?cited by 19
  7. 2024Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learningcited by 13
  8. 2024Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphscited by 10
  9. 2023Modality-Aware Negative Sampling for Multi-modal Knowledge Graph Embeddingcited by 7
  10. 2024Powerformer: A Section-adaptive Transformer for Power Flow Adjustmentcited by 5
  11. 2024DRAK: Unlocking Molecular Insights with Domain-Specific Retrieval-Augmented Knowledge in LLMscited by 3
  12. 2024Root-KGD: A Novel Framework for Root Cause Diagnosis Based on Knowledge Graph and Industrial Datacited by 2
6 more
  1. 2024PORCA: Root Cause Analysis with Partially Observed Datacited by 2
  2. 2025Boosting Knowledge Graph-based Recommendations through Confidence-Aware Augmentation with Large Language Modelscited by 2
  3. 2023CausE: Towards Causal Knowledge Graph Embeddingcited by 1
  4. 2026TeleDoCTR: Domain-Specific and Contextual Troubleshooting for Telecommunicationscited by 1
  5. 2025Collaboration of Fusion and Independence: Hypercomplex-driven Robust Multi-Modal Knowledge Graph Completioncited by 0
  6. 2026SolarGPT-QA: A Domain-Adaptive Large Language Model for Educational Question Answering in Space Weather and Heliophysicscited by 0