# pix2pockets: Shot Suggestions in 8-Ball Pool from a Single Image in the Wild

arXiv: https://arxiv.org/abs/2504.12045 · HTML: https://mlsanity.com/paper/2504.12045/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 18 of its 38 references that are in the map.
- First posted on arXiv 16 Apr 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 18 of 18)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [A Closer Look at Invalid Action Masking in Policy Gradient Algorithms](https://mlsanity.com/paper/2006.14171/) (2020)
- [SoccerNet: A Scalable Dataset for Action Spotting in Soccer Videos](https://mlsanity.com/paper/1804.04527/) (2018)
- [Control of Memory, Active Perception, and Action in Minecraft](https://mlsanity.com/paper/1605.09128/) (2016)
- [Counter-Strike Deathmatch with Large-Scale Behavioural Cloning](https://mlsanity.com/paper/2104.04258/) (2021)
- [Game Plan: What AI can do for Football, and What Football can do for AI](https://mlsanity.com/paper/2011.09192/) (2020)
- [Detecting events and key actors in multi-person videos](https://mlsanity.com/paper/1511.02917/) (2015)
- [Cracking the Black Box: Distilling Deep Sports Analytics](https://mlsanity.com/paper/2006.04551/) (2020)
- [Improving Object Detection Quality in Football Through Super-Resolution Techniques](https://mlsanity.com/paper/2402.00163/) (2024)
- [Deep Reinforcement Learning with Model Learning and Monte Carlo Tree Search in Minecraft](https://mlsanity.com/paper/1803.08456/) (2018)

## Built on by (most cited first, 2 of 2)

- [Chess on Ice: Curling Tactical Decision-Making via Backward Induction and Deep Reinforcement Learning](https://mlsanity.com/paper/2608.02379/) (2026)
- [Rule-Constrained Assignment for Cue-Ball Identification in Broadcast Snooker](https://mlsanity.com/paper/2609.23450/) (2026)

## Abstract (arXiv)

Computer vision models have seen increased usage in sports, and reinforcement learning (RL) is famous for beating humans in strategic games such as Chess and Go. In this paper, we are interested in building upon these advances and examining the game of classic 8-ball pool. We introduce pix2pockets, a foundation for an RL-assisted pool coach. Given a single image of a pool table, we first aim to detect the table and the balls and then propose the optimal shot suggestion. For the first task, we build a dataset with 195 diverse images where we manually annotate all balls and table dots, leading to 5748 object segmentation masks. For the second task, we build a standardized RL environment that allows easy development and benchmarking of any RL algorithm. Our object detection model yields an AP50 of 91.2 while our ball location pipeline obtains an error of only 0.4 cm. Furthermore, we compare standard RL algorithms to set a baseline for the shot suggestion task and we show that all of them fail to pocket all balls without making a foul move. We also present a simple baseline that achieves a per-shot success rate of 94.7% and clears a full game in a single turn 30% of the time.