# Factor Augmented Supervised Learning with Text Embeddings

arXiv: https://arxiv.org/abs/2508.06548 · HTML: https://mlsanity.com/paper/2508.06548/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 67 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts](https://mlsanity.com/paper/1307.5336/) (2013)
- [Deep Semi-Supervised Anomaly Detection](https://mlsanity.com/paper/1906.02694/) (2019)
- [ADBench: Anomaly Detection Benchmark](https://mlsanity.com/paper/2206.09426/) (2022)
- [Large Language Models in Finance: A Survey](https://mlsanity.com/paper/2311.10723/) (2023)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [A Study of Generative Large Language Model for Medical Research and Healthcare](https://mlsanity.com/paper/2305.13523/) (2023)
- [Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models](https://mlsanity.com/paper/2304.07619/) (2023)
- [Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence](https://mlsanity.com/paper/2004.03974/) (2020)
- [Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data](https://mlsanity.com/paper/2407.04491/) (2024)
- [A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges](https://mlsanity.com/paper/2406.11903/) (2024)
- [The Kolmogorov-Arnold representation theorem revisited](https://mlsanity.com/paper/2007.15884/) (2020)
- [Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Review](https://mlsanity.com/paper/2402.10350/) (2024)
- [On the Dimensionality of Word Embedding](https://mlsanity.com/paper/1812.04224/) (2018)
- [Using SVD for Topic Modeling](https://mlsanity.com/paper/1704.07016/) (2017)
- [Estimating Number of Factors by Adjusted Eigenvalues Thresholding](https://mlsanity.com/paper/1909.10710/) (2019)
- [FinEntity: Entity-level Sentiment Classification for Financial Texts](https://mlsanity.com/paper/2310.12406/) (2023)
- [Sufficient Forecasting Using Factor Models](https://mlsanity.com/paper/1505.07414/) (2015)
- [Inverse Moment Methods for Sufficient Forecasting using High-Dimensional Predictors](https://mlsanity.com/paper/1705.00395/) (2017)
- [Single Training Dimension Selection for Word Embedding with PCA](https://mlsanity.com/paper/1909.01761/) (2019)
- [Factor Augmented Tensor-on-Tensor Neural Networks](https://mlsanity.com/paper/2405.19610/) (2024)
- [Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings](https://mlsanity.com/paper/2403.14001/) (2024)
- [When Dimensionality Hurts: The Role of LLM Embedding Compression for Noisy Regression Tasks](https://mlsanity.com/paper/2502.02199/) (2025)
- [PCA-RAG: Principal Component Analysis for Efficient Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.08386/) (2025)

## Built on by (most cited first, 1 of 1)

- [Factor Augmented High-Dimensional SGD](https://mlsanity.com/paper/2605.19291/) (2026)

## Abstract (arXiv)

Large language models (LLMs) generate text embeddings from text data, producing vector representations that capture the semantic meaning and contextual relationships of words. However, the high dimensionality of these embeddings often impedes efficiency and drives up computational cost in downstream tasks. To address this, we propose AutoEncoder-Augmented Learning with Text (AEALT), a supervised, factor-augmented framework that incorporates dimension reduction directly into pre-trained LLM workflows. First, we extract embeddings from text documents; next, we pass them through a supervised augmented autoencoder to learn low-dimensional, task-relevant latent factors. By modeling the nonlinear structure of complex embeddings, AEALT outperforms conventional deep-learning approaches that rely on raw embeddings. We validate its broad applicability with extensive experiments on classification, anomaly detection, and prediction tasks using multiple real-world public datasets. Numerical results demonstrate that AEALT yields substantial gains over both vanilla embeddings and several standard dimension reduction methods.