# Tele-Knowledge Pre-training for Fault Analysis

arXiv: https://arxiv.org/abs/2210.11298 · HTML: https://mlsanity.com/paper/2210.11298/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 54 references that are in the map.
- First posted on arXiv 20 Oct 2022.

## Builds on (most cited first, 30 of 43)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://mlsanity.com/paper/1810.11921/) (2018)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://mlsanity.com/paper/2107.07653/) (2021)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction](https://mlsanity.com/paper/2104.07650/) (2021)
- [LogBERT: Log Anomaly Detection via BERT](https://mlsanity.com/paper/2103.04475/) (2021)
- [Should You Mask 15% in Masked Language Modeling?](https://mlsanity.com/paper/2202.08005/) (2022)
- [TABBIE: Pretrained Representations of Tabular Data](https://mlsanity.com/paper/2105.02584/) (2021)
- [Neural Photo Editing with Introspective Adversarial Networks](https://mlsanity.com/paper/1609.07093/) (2016)
- [StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding](https://mlsanity.com/paper/1908.04577/) (2019)
- [TUTA: Tree-based Transformers for Generally Structured Table Pre-training](https://mlsanity.com/paper/2010.12537/) (2020)

## Built on by (most cited first, 18 of 18)

- [Time-MQA: Time Series Multi-Task Question Answering with Context Enhancement](https://mlsanity.com/paper/2503.01875/) (2025)
- [Making Large Language Models Perform Better in Knowledge Graph Completion](https://mlsanity.com/paper/2310.06671/) (2023)
- [OceanGPT: A Large Language Model for Ocean Science Tasks](https://mlsanity.com/paper/2310.02031/) (2023)
- [TelecomGPT: A Framework to Build Telecom-Specfic Large Language Models](https://mlsanity.com/paper/2407.09424/) (2024)
- [Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering](https://mlsanity.com/paper/2311.06503/) (2023)
- [When Do Program-of-Thoughts Work for Reasoning?](https://mlsanity.com/paper/2308.15452/) (2023)
- [Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning](https://mlsanity.com/paper/2405.16869/) (2024)
- [Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphs](https://mlsanity.com/paper/2404.00942/) (2024)
- [Modality-Aware Negative Sampling for Multi-modal Knowledge Graph Embedding](https://mlsanity.com/paper/2304.11618/) (2023)
- [Powerformer: A Section-adaptive Transformer for Power Flow Adjustment](https://mlsanity.com/paper/2401.02771/) (2024)
- [DRAK: Unlocking Molecular Insights with Domain-Specific Retrieval-Augmented Knowledge in LLMs](https://mlsanity.com/paper/2406.18535/) (2024)
- [Root-KGD: A Novel Framework for Root Cause Diagnosis Based on Knowledge Graph and Industrial Data](https://mlsanity.com/paper/2406.13664/) (2024)
- [PORCA: Root Cause Analysis with Partially Observed Data](https://mlsanity.com/paper/2407.05869/) (2024)
- [Boosting Knowledge Graph-based Recommendations through Confidence-Aware Augmentation with Large Language Models](https://mlsanity.com/paper/2502.03715/) (2025)
- [CausE: Towards Causal Knowledge Graph Embedding](https://mlsanity.com/paper/2307.11610/) (2023)
- [TeleDoCTR: Domain-Specific and Contextual Troubleshooting for Telecommunications](https://mlsanity.com/paper/2601.00691/) (2026)
- [Collaboration of Fusion and Independence: Hypercomplex-driven Robust Multi-Modal Knowledge Graph Completion](https://mlsanity.com/paper/2509.23714/) (2025)
- [SolarGPT-QA: A Domain-Adaptive Large Language Model for Educational Question Answering in Space Weather and Heliophysics](https://mlsanity.com/paper/2601.12131/) (2026)

## Abstract (arXiv)

In this work, we share our experience on tele-knowledge pre-training for fault analysis, a crucial task in telecommunication applications that requires a wide range of knowledge normally found in both machine log data and product documents. To organize this knowledge from experts uniformly, we propose to create a Tele-KG (tele-knowledge graph). Using this valuable data, we further propose a tele-domain language pre-training model TeleBERT and its knowledge-enhanced version, a tele-knowledge re-training model KTeleBERT. which includes effective prompt hints, adaptive numerical data encoding, and two knowledge injection paradigms. Concretely, our proposal includes two stages: first, pre-training TeleBERT on 20 million tele-related corpora, and then re-training it on 1 million causal and machine-related corpora to obtain KTeleBERT. Our evaluation on multiple tasks related to fault analysis in tele-applications, including root-cause analysis, event association prediction, and fault chain tracing, shows that pre-training a language model with tele-domain data is beneficial for downstream tasks. Moreover, the KTeleBERT re-training further improves the performance of task models, highlighting the effectiveness of incorporating diverse tele-knowledge into the model.