mlsanityData to 14 Oct 2026

Pretrained deep models outperform GBDTs in Learning-To-Rank under label scarcity

arXiv 2308.00177 ↗First posted 31 Jul 2023cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

On tabular data, a significant body of literature has shown that current deep learning (DL) models perform at best similarly to Gradient Boosted Decision Trees (GBDTs), while significantly underperforming them on outlier data. However, these works often study idealized problem settings which may fail to capture complexities of real-world scenarios. We identify a natural tabular data setting where DL models can outperform GBDTs: tabular Learning-to-Rank (LTR) under label scarcity. Tabular LTR applications, including search and recommendation, often have an abundance of unlabeled data, and scarce labeled data. We show that DL rankers can utilize unsupervised pretraining to exploit this unlabeled data. In extensive experiments over both public and proprietary datasets, we show that pretrained DL rankers consistently outperform GBDT rankers on ranking metrics -- sometimes by as much as 38% -- both overall and on outliers.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 91 of its 150 references that are in the map. First posted on arXiv 31 Jul 2023.

Builds on

91 of 150 references.

References · 91

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
38 more
  1. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
  2. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  3. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  4. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  5. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 920
  6. 2019On the Convergence of Adam and Beyondcited by 841
  7. 2016Wide & Deep Learning for Recommender Systemscited by 747
  8. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  9. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  10. 2017DeepFM: A Factorization-Machine based Neural Network for CTR Predictioncited by 676
  11. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
  12. 2020Contrastive Learning with Hard Negative Samplescited by 628
  13. 2021Tabular Data: Deep Learning is Not All You Needcited by 535
  14. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
  15. 2021Deep Neural Networks and Tabular Data: A Surveycited by 491
  16. 2015Deep Metric Learning via Lifted Structured Feature Embeddingcited by 449
  17. 2016Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learningcited by 441
  18. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
  19. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  20. 2020DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systemscited by 327
  21. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 319
  22. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
  23. 2017Sampling Matters in Deep Embedding Learningcited by 285
  24. 2018xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systemscited by 278
  25. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 277
  26. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 267
  27. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  28. 2017Distilling a Neural Network Into a Soft Decision Treecited by 236
  29. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 223
  30. 2019Neural Oblivious Decision Ensembles for Deep Learning on Tabular Datacited by 203
  31. 2019Multi-Stage Document Ranking with BERTcited by 201
  32. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  33. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 156
  34. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  35. 2016Product-based Neural Networks for User Response Predictioncited by 145
  36. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  37. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 125
  38. 2013Introducing LETOR 4.0 Datasetscited by 121

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025RewardRank: Optimizing True Learning-to-Rank Utilitycited by 0