# Enhancing Risk Assessment in Transformers with Loss-at-Risk Functions

arXiv: https://arxiv.org/abs/2411.02558 · HTML: https://mlsanity.com/paper/2411.02558/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 60 references that are in the map.
- First posted on arXiv 4 Nov 2024.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [FinGPT: Open-Source Financial Large Language Models](https://mlsanity.com/paper/2306.06031/) (2023)
- [Large Language Models in Finance: A Survey](https://mlsanity.com/paper/2311.10723/) (2023)
- [Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach](https://mlsanity.com/paper/1506.02188/) (2015)
- [Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation](https://mlsanity.com/paper/2105.08919/) (2021)
- [Optimizing the CVaR via Sampling](https://mlsanity.com/paper/1404.3862/) (2014)
- [Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search](https://mlsanity.com/paper/2303.06573/) (2023)
- [ConvGQR: Generative Query Reformulation for Conversational Search](https://mlsanity.com/paper/2305.15645/) (2023)
- [Exploring the Encoding Layer and Loss Function in End-to-End Speaker and Language Recognition System](https://mlsanity.com/paper/1804.05160/) (2018)
- [RATT: A Thought Structure for Coherent and Correct LLM Reasoning](https://mlsanity.com/paper/2406.02746/) (2024)
- [A Comprehensive Survey of Regression Based Loss Functions for Time Series Forecasting](https://mlsanity.com/paper/2211.02989/) (2022)
- [Self-optimizing Feature Generation via Categorical Hashing Representation and Hierarchical Reinforcement Crossing](https://mlsanity.com/paper/2309.04612/) (2023)
- [Dynamic and Adaptive Feature Generation with LLM](https://mlsanity.com/paper/2406.03505/) (2024)
- [Prototypical Reward Network for Data-Efficient RLHF](https://mlsanity.com/paper/2406.06606/) (2024)
- [MMREC: LLM Based Multi-Modal Recommender System](https://mlsanity.com/paper/2408.04211/) (2024)
- [A Survey of Conversational Search](https://mlsanity.com/paper/2410.15576/) (2024)
- [Feature Selection as Deep Sequential Generative Learning](https://mlsanity.com/paper/2403.03838/) (2024)
- [Research on Autonomous Driving Decision-making Strategies based Deep Reinforcement Learning](https://mlsanity.com/paper/2408.03084/) (2024)
- [Training a Task-Specific Image Reconstruction Loss](https://mlsanity.com/paper/2103.14616/) (2021)
- [Data Imputation using Large Language Model to Accelerate Recommendation System](https://mlsanity.com/paper/2407.10078/) (2024)
- [Title2Event: Benchmarking Open Event Extraction with a Large-scale Chinese Title Dataset](https://mlsanity.com/paper/2211.00869/) (2022)
- [Neuro-Symbolic Embedding for Short and Effective Feature Selection via Autoregressive Generation](https://mlsanity.com/paper/2404.17157/) (2024)
- [TFWT: Tabular Feature Weighting with Transformer](https://mlsanity.com/paper/2405.08403/) (2024)
- [Machine Learning-Based Research on the Adaptability of Adolescents to Online Education](https://mlsanity.com/paper/2408.16849/) (2024)
- [Learning A Task-Specific Deep Architecture For Clustering](https://mlsanity.com/paper/1509.00151/) (2015)
- [Predicting NVIDIA's Next-Day Stock Price: A Comparative Analysis of LSTM, MLP, ARIMA, and ARIMA-GARCH Models](https://mlsanity.com/paper/2405.08284/) (2024)

## Built on by (most cited first, 2 of 2)

- [A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents](https://mlsanity.com/paper/2506.23844/) (2025)
- [CT-PatchTST: Channel-Time Patch Time-Series Transformer for Long-Term Renewable Energy Forecasting](https://mlsanity.com/paper/2501.08620/) (2025)

## Abstract (arXiv)

In the financial field, precise risk assessment tools are essential for decision-making. Recent studies have challenged the notion that traditional network loss functions like Mean Square Error (MSE) are adequate, especially under extreme risk conditions that can lead to significant losses during market upheavals. Transformers and Transformer-based models are now widely used in financial forecasting according to their outstanding performance in time-series-related predictions. However, these models typically lack sensitivity to extreme risks and often underestimate great financial losses. To address this problem, we introduce a novel loss function, the Loss-at-Risk, which incorporates Value at Risk (VaR) and Conditional Value at Risk (CVaR) into Transformer models. This integration allows Transformer models to recognize potential extreme losses and further improves their capability to handle high-stakes financial decisions. Moreover, we conduct a series of experiments with highly volatile financial datasets to demonstrate that our Loss-at-Risk function improves the Transformers' risk prediction and management capabilities without compromising their decision-making accuracy or efficiency. The results demonstrate that integrating risk-aware metrics during training enhances the Transformers' risk assessment capabilities while preserving their core strengths in decision-making and reasoning across diverse scenarios.