# Learning to Bid Long-Term: Multi-Agent Reinforcement Learning with Long-Term and Sparse Reward in Repeated Auction Games

arXiv: https://arxiv.org/abs/2204.02268 · HTML: https://mlsanity.com/paper/2204.02268/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 31 references that are in the map.
- First posted on arXiv 5 Apr 2022.

## Builds on (most cited first, 7 of 7)

- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016)
- [Large-Scale Study of Curiosity-Driven Learning](https://mlsanity.com/paper/1808.04355/) (2018)
- [Training Very Deep Networks](https://mlsanity.com/paper/1507.06228/) (2015)
- [Optimizing Agent Behavior over Long Time Scales by Transporting Value](https://mlsanity.com/paper/1810.06721/) (2018)
- [Optimal Bidding in Repeated Wireless Spectrum Auctions with Budget Constraints](https://mlsanity.com/paper/1608.07357/) (2016)
- [Learning from Delayed Outcomes via Proxies with Applications to Recommender Systems](https://mlsanity.com/paper/1807.09387/) (2018)

## Built on by (most cited first, 3 of 3)

- [Coordinated Dynamic Bidding in Repeated Second-Price Auctions with Budgets](https://mlsanity.com/paper/2306.07709/) (2023)
- [Multi-Agent Reinforcement Learning for Long-Term Network Resource Allocation through Auction: a V2X Application](https://mlsanity.com/paper/2208.04237/) (2022)
- [Multi-Objective Optimization Using Adaptive Distributed Reinforcement Learning](https://mlsanity.com/paper/2403.08879/) (2024)

## Abstract (arXiv)

We propose a multi-agent distributed reinforcement learning algorithm that balances between potentially conflicting short-term reward and sparse, delayed long-term reward, and learns with partial information in a dynamic environment. We compare different long-term rewards to incentivize the algorithm to maximize individual payoff and overall social welfare. We test the algorithm in two simulated auction games, and demonstrate that 1) our algorithm outperforms two benchmark algorithms in a direct competition, with cost to social welfare, and 2) our algorithm's aggressive competitive behavior can be guided with the long-term reward signal to maximize both individual payoff and overall social welfare.