# Neuro-Symbolic Constraint Programming for Structured Prediction

arXiv: https://arxiv.org/abs/2103.17232 · HTML: https://mlsanity.com/paper/2103.17232/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 51 references that are in the map.
- First posted on arXiv 31 Mar 2021.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Input Convex Neural Networks](https://mlsanity.com/paper/1609.07152/) (2016)
- [Neural Probabilistic Logic Programming in DeepProbLog](https://mlsanity.com/paper/1907.08194/) (2019)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [Smart "Predict, then Optimize"](https://mlsanity.com/paper/1710.08005/) (2017)
- [A Semantic Loss Function for Deep Learning with Symbolic Knowledge](https://mlsanity.com/paper/1711.11157/) (2017)
- [Harnessing Deep Neural Networks with Logic Rules](https://mlsanity.com/paper/1603.06318/) (2016)
- [Logic Tensor Networks for Semantic Image Interpretation](https://mlsanity.com/paper/1705.08968/) (2017)
- [Smart Predict-and-Optimize for Hard Combinatorial Optimization Problems](https://mlsanity.com/paper/1911.10092/) (2019)
- [Interior Point Solving for LP-based prediction+optimisation](https://mlsanity.com/paper/2010.13943/) (2020)
- [Relational Deep Reinforcement Learning](https://mlsanity.com/paper/1806.01830/) (2018)
- [From Statistical Relational to Neuro-Symbolic Artificial Intelligence](https://mlsanity.com/paper/2003.08316/) (2020)
- [Structured Prediction Energy Networks](https://mlsanity.com/paper/1511.06350/) (2015)
- [Towards Deep Symbolic Reinforcement Learning](https://mlsanity.com/paper/1609.05518/) (2016)
- [Efficient Generation of Structured Objects with Constrained Adversarial Networks](https://mlsanity.com/paper/2007.13197/) (2020)
- [Learning Linear Programs from Optimal Decisions](https://mlsanity.com/paper/2006.08923/) (2020)
- [Deep Value Networks Learn to Evaluate and Iteratively Refine Structured Outputs](https://mlsanity.com/paper/1703.04363/) (2017)
- [Human-Level Intelligence or Animal-Like Abilities?](https://mlsanity.com/paper/1707.04327/) (2017)
- [Relational Bayesian Networks](https://mlsanity.com/paper/1302.1550/) (2013)
- [Structured Learning Modulo Theories](https://mlsanity.com/paper/1405.1675/) (2014)
- [Deep Inverse Optimization](https://mlsanity.com/paper/1812.00804/) (2018)

## Built on by (most cited first, 11 of 11)

- [Deep Learning with Logical Constraints](https://mlsanity.com/paper/2205.00523/) (2022)
- [A Survey on Neural-symbolic Learning Systems](https://mlsanity.com/paper/2111.08164/) (2021)
- [ROAD-R: The Autonomous Driving Dataset with Logical Requirements](https://mlsanity.com/paper/2210.01597/) (2022)
- [Logic of Differentiable Logics: Towards a Uniform Semantics of DL](https://mlsanity.com/paper/2303.10650/) (2023)
- [Refining neural network predictions using background knowledge](https://mlsanity.com/paper/2206.04976/) (2022)
- [Weakly Supervised Anomaly Detection via Knowledge-Data Alignment](https://mlsanity.com/paper/2402.03785/) (2024)
- [Neuro-Symbolic AI: Explainability, Challenges, and Future Trends](https://mlsanity.com/paper/2411.04383/) (2024)
- [Machine Learning with Requirements: a Manifesto](https://mlsanity.com/paper/2304.03674/) (2023)
- [Exploiting T-norms for Deep Learning in Autonomous Driving](https://mlsanity.com/paper/2402.11362/) (2024)
- [Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignment](https://mlsanity.com/paper/2602.14844/) (2026)
- [Bridging the Gap: Representation Spaces in Neuro-Symbolic AI](https://mlsanity.com/paper/2411.04393/) (2024)

## Abstract (arXiv)

We propose Nester, a method for injecting neural networks into constrained structured predictors. The job of the neural network(s) is to compute an initial, raw prediction that is compatible with the input data but does not necessarily satisfy the constraints. The structured predictor then builds a structure using a constraint solver that assembles and corrects the raw predictions in accordance with hard and soft constraints. In doing so, Nester takes advantage of the features of its two components: the neural network learns complex representations from low-level data while the constraint programming component reasons about the high-level properties of the prediction task. The entire architecture can be trained in an end-to-end fashion. An empirical evaluation on handwritten equation recognition shows that Nester achieves better performance than both the neural network and the constrained structured predictor on their own, especially when training examples are scarce, while scaling to more complex problems than other neuro-programming approaches. Nester proves especially useful to reduce errors at the semantic level of the problem, which is particularly challenging for neural network architectures.Sub