# DOM-LM: Learning Generalizable Representations for HTML Documents

arXiv: https://arxiv.org/abs/2201.10608 · HTML: https://mlsanity.com/paper/2201.10608/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 25 Jan 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [WebSRC: A Dataset for Web-Based Structural Reading Comprehension](https://mlsanity.com/paper/2101.09465/) (2021)
- [Language-Agnostic Representation Learning of Source Code from Structure and Context](https://mlsanity.com/paper/2103.11318/) (2021)
- [Few-Shot Question Answering by Pretraining Span Selection](https://mlsanity.com/paper/2101.00438/) (2021)
- [Modeling Relation Paths for Representation Learning of Knowledge Bases](https://mlsanity.com/paper/1506.00379/) (2015)
- [MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding](https://mlsanity.com/paper/2110.08518/) (2021)
- [Universal Representation Learning of Knowledge Bases by Jointly Embedding Instances and Ontological Concepts](https://mlsanity.com/paper/2103.08115/) (2021)
- [ZeroShotCeres: Zero-Shot Relation Extraction from Semi-Structured Webpages](https://mlsanity.com/paper/2005.07105/) (2020)
- [FreeDOM: A Transferable Neural Architecture for Structured Information Extraction on Web Documents](https://mlsanity.com/paper/2010.10755/) (2020)
- [ReasonBERT: Pre-trained to Reason with Distant Supervision](https://mlsanity.com/paper/2109.04912/) (2021)
- [Simplified DOM Trees for Transferable Attribute Extraction from the Web](https://mlsanity.com/paper/2101.02415/) (2021)
- [Fonduer: Knowledge Base Construction from Richly Formatted Data](https://mlsanity.com/paper/1703.05028/) (2017)
- [CERES: Distantly Supervised Relation Extraction from the Semi-Structured Web](https://mlsanity.com/paper/1804.04635/) (2018)

## Built on by (most cited first, 24 of 24)

- [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents](https://mlsanity.com/paper/2306.16527/) (2023)
- [WebLINX: Real-World Website Navigation with Multi-Turn Dialogue](https://mlsanity.com/paper/2402.05930/) (2024)
- [Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes](https://mlsanity.com/paper/2304.09433/) (2023)
- [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023)
- [Longhorn: State Space Models are Amortized Online Learners](https://mlsanity.com/paper/2407.14207/) (2024)
- [Embedding-based Retrieval with LLM for Effective Agriculture Information Extracting from Unstructured Data](https://mlsanity.com/paper/2308.03107/) (2023)
- [LLM/Agent-as-Data-Analyst: A Survey](https://mlsanity.com/paper/2509.23988/) (2025)
- [XDoc: Unified Pre-training for Cross-Format Document Understanding](https://mlsanity.com/paper/2210.02849/) (2022)
- [From Grounding to Planning: Benchmarking Bottlenecks in Web Agents](https://mlsanity.com/paper/2409.01927/) (2024)
- [SEED: Domain-Specific Data Curation With Large Language Models](https://mlsanity.com/paper/2310.00749/) (2023)
- [On Web-based Visual Corpus Construction for Visual Document Understanding](https://mlsanity.com/paper/2211.03256/) (2022)
- [SAGEO Arena: A Realistic Environment for Evaluating Search-Augmented Generative Engine Optimization](https://mlsanity.com/paper/2602.12187/) (2026)
- [Hierarchical Multimodal Pre-training for Visually Rich Webpage Understanding](https://mlsanity.com/paper/2402.18262/) (2024)
- [WebRPG: Automatic Web Rendering Parameters Generation for Visual Presentation](https://mlsanity.com/paper/2407.15502/) (2024)
- [PLAtE: A Large-scale Dataset for List Page Web Extraction](https://mlsanity.com/paper/2205.12386/) (2022)
- [Enhancing Mobile "How-to" Queries with Automated Search Results Verification and Reranking](https://mlsanity.com/paper/2404.08860/) (2024)
- [ALLOY: Generating Reusable Agent Workflows from User Demonstration](https://mlsanity.com/paper/2510.10049/) (2025)
- [Combining Language and Graph Models for Semi-structured Information Extraction on the Web](https://mlsanity.com/paper/2402.14129/) (2024)
- [From Context to Action: Analysis of the Impact of State Representation and Context on the Generalization of Multi-Turn Web Navigation Agents](https://mlsanity.com/paper/2410.23555/) (2024)
- [Multilingual Attribute Extraction from News Web Pages](https://mlsanity.com/paper/2502.02167/) (2025)
- [Knowledge Extraction on Semi-Structured Content: Does It Remain Relevant for Question Answering in the Era of LLMs?](https://mlsanity.com/paper/2509.25107/) (2025)
- [URL-BERT: Training Webpage Representations via Social Media Engagements](https://mlsanity.com/paper/2310.16303/) (2023)
- [ReFinE: Streamlining UI Mockup Iteration with Research Findings](https://mlsanity.com/paper/2604.04353/) (2026)
- [news-crawler-LM: A Small Long-Context Model For High-Quality News Crawling](https://mlsanity.com/paper/2607.21284/) (2026)

## Abstract (arXiv)

HTML documents are an important medium for disseminating information on the Web for human consumption. An HTML document presents information in multiple text formats including unstructured text, structured key-value pairs, and tables. Effective representation of these documents is essential for machine understanding to enable a wide range of applications, such as Question Answering, Web Search, and Personalization. Existing work has either represented these documents using visual features extracted by rendering them in a browser, which is typically computationally expensive, or has simply treated them as plain text documents, thereby failing to capture useful information presented in their HTML structure. We argue that the text and HTML structure together convey important semantics of the content and therefore warrant a special treatment for their representation learning. In this paper, we introduce a novel representation learning approach for web pages, dubbed DOM-LM, which addresses the limitations of existing approaches by encoding both text and DOM tree structure with a transformer-based encoder and learning generalizable representations for HTML documents via self-supervised pre-training. We evaluate DOM-LM on a variety of webpage understanding tasks, including Attribute Extraction, Open Information Extraction, and Question Answering. Our extensive experiments show that DOM-LM consistently outperforms all baselines designed for these tasks. In particular, DOM-LM demonstrates better generalization performance both in few-shot and zero-shot settings, making it attractive for making it suitable for real-world application settings with limited labeled data.