# Pretrained Domain-Specific Language Model for General Information Retrieval Tasks in the AEC Domain

arXiv: https://arxiv.org/abs/2203.04729 · HTML: https://mlsanity.com/paper/2203.04729/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 55 references that are in the map.
- First posted on arXiv 9 Mar 2022.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [A Survey on Deep Transfer Learning](https://mlsanity.com/paper/1808.01974/) (2018)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [Recurrent Neural Network for Text Classification with Multi-Task Learning](https://mlsanity.com/paper/1605.05101/) (2016)
- [A Comparison of Word Embeddings for the Biomedical Natural Language Processing](https://mlsanity.com/paper/1802.00400/) (2018)
- [Low Resource Recognition and Linking of Biomedical Concepts from a Large Ontology](https://mlsanity.com/paper/2101.10587/) (2021)

## Built on by (most cited first, 12 of 12)

- [Open-world Story Generation with Structured Knowledge Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2212.04634/) (2022)
- [Translating Regulatory Clauses into Executable Codes for Building Design Checking via Large Language Model Driven Function Matching and Composing](https://mlsanity.com/paper/2308.08728/) (2023)
- [Automating construction contract review using knowledge graph-enhanced large language models](https://mlsanity.com/paper/2309.12132/) (2023)
- [A Text Classification-Based Approach for Evaluating and Enhancing the Machine Interpretability of Building Codes](https://mlsanity.com/paper/2309.14374/) (2023)
- [QuakeBERT: Accurate Classification of Social Media Texts for Rapid Earthquake Impact Assessment](https://mlsanity.com/paper/2405.06684/) (2024)
- [Natural Language Instructions for Intuitive Human Interaction with Robotic Assistants in Field Construction Work](https://mlsanity.com/paper/2307.04195/) (2023)
- [A Zipf's Law-based Text Generation Approach for Addressing Imbalance in Entity Extraction](https://mlsanity.com/paper/2205.12636/) (2022)
- [A new hazard event classification model via deep learning and multifractal](https://mlsanity.com/paper/2209.05263/) (2022)
- [BIGNet: Pretrained Graph Neural Network for Embedding Semantic, Spatial, and Topological Data in BIM Models](https://mlsanity.com/paper/2509.11104/) (2025)
- [Yes, DLGM! A novel hierarchical model for hazard classification](https://mlsanity.com/paper/2209.04576/) (2022)
- [Extracting Structured Requirements from Unstructured Building Technical Specifications for Building Information Modeling](https://mlsanity.com/paper/2508.13833/) (2025)
- [Domain-specific Pretraining Profile and Transformer Performance: Evidence from Modeling Digital Pragmatics in Arabic-English Code-switching](https://mlsanity.com/paper/2609.14571/) (2026)

## Abstract (arXiv)

As an essential task for the architecture, engineering, and construction (AEC) industry, information retrieval (IR) from unstructured textual data based on natural language processing (NLP) is gaining increasing attention. Although various deep learning (DL) models for IR tasks have been investigated in the AEC domain, it is still unclear how domain corpora and domain-specific pretrained DL models can improve performance in various IR tasks. To this end, this work systematically explores the impacts of domain corpora and various transfer learning techniques on the performance of DL models for IR tasks and proposes a pretrained domain-specific language model for the AEC domain. First, both in-domain and close-domain corpora are developed. Then, two types of pretrained models, including traditional wording embedding models and BERT-based models, are pretrained based on various domain corpora and transfer learning strategies. Finally, several widely used DL models for IR tasks are further trained and tested based on various configurations and pretrained models. The result shows that domain corpora have opposite effects on traditional word embedding models for text classification and named entity recognition tasks but can further improve the performance of BERT-based models in all tasks. Meanwhile, BERT-based models dramatically outperform traditional methods in all IR tasks, with maximum improvements of 5.4% and 10.1% in the F1 score, respectively. This research contributes to the body of knowledge in two ways: 1) demonstrating the advantages of domain corpora and pretrained DL models and 2) opening the first domain-specific dataset and pretrained language model for the AEC domain, to the best of our knowledge. Thus, this work sheds light on the adoption and application of pretrained models in the AEC domain.