mlsanityData to 7 Oct 2026

TextGNN: Improving Text Encoder via Graph Neural Network in Sponsored Search

arXiv 2101.06323 ↗First posted 15 Jan 2021cs.CL cs.LGcited by 54

Not on the map. Why?

Abstract (arXiv)

Text encoders based on C-DSSM or transformers have demonstrated strong performance in many Natural Language Processing (NLP) tasks. Low latency variants of these models have also been developed in recent years in order to apply them in the field of sponsored search which has strict computational constraints. However these models are not the panacea to solve all the Natural Language Understanding (NLU) challenges as the pure semantic information in the data is not sufficient to fully identify the user intents. We propose the TextGNN model that naturally extends the strong twin tower structured encoders with the complementary graph information from user historical behaviors, which serves as a natural guide to help us better understand the intents and hence generate better language representations. The model inherits all the benefits of twin tower models such as C-DSSM and TwinBERT so that it can still be used in the low latency environment while achieving a significant performance gain than the strong encoder-only counterpart baseline models in both offline evaluations and online production system. In offline experiments, the model achieves a 0.14% overall increase in ROC-AUC with a 1% increased accuracy for long-tail low-frequency Ads, and in the online A/B testing, the model shows a 2.03% increase in Revenue Per Mille with a 2.32% decrease in Ad defect rate.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 16 of its 30 references that are in the map. First posted on arXiv 15 Jan 2021.

Builds on

16 of 30 references.

References · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2017Graph Attention Networkscited by 8,854
  8. 2017Inductive Representation Learning on Large Graphscited by 6,284
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,571
  12. 2020Embedding-based Retrieval in Facebook Searchcited by 155
4 more
  1. 2020PinnerSage: Multi-Modal User Embedding Framework for Recommendations at Pinterestcited by 75
  2. 2017Neural Ranking Models with Weak Supervisioncited by 61
  3. 2019Fi-GNN: Modeling Feature Interactions via Graph Neural Networks for CTR Predictioncited by 54
  4. 2020TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrievalcited by 24

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 6; 2023: 16; 2024: 18; 2025: 13.

54 papers.

Sort:

By area:

  1. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  2. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 190
  3. 2022Learning on Large-scale Text-attributed Graphs via Variational Inferencecited by 153
  4. 2021GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graphcited by 143
  5. 2023Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learningcited by 132
  6. 2023SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learningcited by 73
  7. 2023Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applicationscited by 33
  8. 2024Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrievalcited by 31
  9. 2023Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphscited by 29
  10. 2023GRENADE: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphscited by 28
  11. 2023Large Language Models as Topological Structure Enhancers for Text-Attributed Graphscited by 27
  12. 2022Heterformer: Transformer-based Deep Node Representation Learning on Heterogeneous Text-Rich Networkscited by 24
38 more
  1. 2023Edgeformers: Graph-Empowered Transformers for Representation Learning on Textual-Edge Networkscited by 23
  2. 2023Leveraging Large Language Models for Node Generation in Few-Shot Learning on Text-Attributed Graphscited by 23
  3. 2024Distilling Large Language Models for Text-Attributed Graph Learningcited by 20
  4. 2025Missing Target-Relevant Information Prediction with World Model for Accurate Zero-Shot Composed Image Retrievalcited by 17
  5. 2023Pretraining Language Models with Text-Attributed Heterogeneous Graphscited by 15
  6. 2024Can Graph Neural Networks Learn Language with Extremely Weak Text Supervision?cited by 13
  7. 2023TouchUp-G: Improving Feature Representation through Graph-Centric Finetuningcited by 12
  8. 2024Pre-Training and Prompting for Few-Shot Node Classification on Text-Attributed Graphscited by 11
  9. 2024Similarity-based Neighbor Selection for Graph LLMscited by 10
  10. 2025Can LLMs Find Fraudsters? Multi-level LLM Enhanced Graph Fraud Detectioncited by 10
  11. 2022Efficient and effective training of language and graph neural network modelscited by 9
  12. 2025Model Generalization on Text Attribute Graphs: Principles with Large Language Modelscited by 8
  13. 2023Label Deconvolution for Node Representation Learning on Large-scale Attributed Graphs against Learning Biascited by 7
  14. 2024HyperBERT: Mixing Hypergraph-Aware Layers with Language Models for Node Classification on Text-Attributed Hypergraphscited by 7
  15. 2025Contrastive Cross-Course Knowledge Tracing via Concept Graph Guided Knowledge Transfercited by 7
  16. 2024Enhancing Collaborative Semantics of Language Model-Driven Recommendations via Graph-Aware Learningcited by 5
  17. 2024Investigating Instruction Tuning Large Language Models on Graphscited by 5
  18. 2024Towards Boosting LLMs-driven Relevance Modeling with Progressive Retrieved Behavior-augmented Promptingcited by 5
  19. 2025Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applicationscited by 5
  20. 2023Beyond Semantics: Learning a Behavior Augmented Relevance Model with Self-supervised Learningcited by 4
  21. 2022TEDL: A Two-stage Evidential Deep Learning Method for Classification Uncertainty Quantificationcited by 3
  22. 2023Align before Search: Aligning Ads Image to Text for Accurate Cross-Modal Sponsored Searchcited by 3
  23. 2024Unleashing the Potential of Text-attributed Graphs: Automatic Relation Decomposition via Large Language Modelscited by 3
  24. 2024Hierarchical Compression of Text-Rich Graphs via Large Language Modelscited by 3
  25. 2024All Against Some: Efficient Integration of Large Language Models for Message Passing in Graph Neural Networkscited by 3
  26. 2024Verbalized Graph Representation Learning: A Fully Interpretable Graph Model Based on Large Language Models Throughout the Entire Processcited by 3
  27. 2024Large Language Models Meet Graph Neural Networks: A Perspective of Graph Miningcited by 3
  28. 2025Integrating Structural and Semantic Signals in Text-Attributed Graphs with BiGTexcited by 3
  29. 2023Enhancing Dynamic Image Advertising with Vision-Language Pre-trainingcited by 2
  30. 2025HierPromptLM: A Pure PLM-based Framework for Representation Learning on Heterogeneous Text-rich Networkscited by 2
  31. 2025A Scalable and Efficient Signal Integration System for Job Matchingcited by 2
  32. 2022MPClan: Protocol Suite for Privacy-Conscious Computationscited by 1
  33. 2024Graph Regularized Encoder Training for Extreme Classificationcited by 1
  34. 2024Node Level Graph Autoencoder: Unified Pretraining for Textual Graph Learningcited by 1
  35. 2024Can Large Language Models Act as Ensembler for Multi-GNNs?cited by 1
  36. 2025Preference-driven Knowledge Distillation for Few-shot Node Classificationcited by 1
  37. 2025Graph-Based Alternatives to LLMs for Human Simulationcited by 1
  38. 2022Text Enriched Sparse Hyperbolic Graph Convolutional Networkscited by 0

50 most cited of 54 All 54 (JSON)