# On Regularization and Inference with Label Constraints

arXiv: https://arxiv.org/abs/2307.03886 · HTML: https://mlsanity.com/paper/2307.03886/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 35 references that are in the map.
- First posted on arXiv 8 Jul 2023.

## Builds on (most cited first, 14 of 14)

- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [A Semantic Loss Function for Deep Learning with Symbolic Knowledge](https://mlsanity.com/paper/1711.11157/) (2017)
- [NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics](https://mlsanity.com/paper/2112.08726/) (2021)
- [Harnessing Deep Neural Networks with Logic Rules](https://mlsanity.com/paper/1603.06318/) (2016)
- [A Multi-Axis Annotation Scheme for Event Temporal Relations](https://mlsanity.com/paper/1804.07828/) (2018)
- [A Logic-Driven Framework for Consistency of Neural Models](https://mlsanity.com/paper/1909.00126/) (2019)
- [Joint Reasoning for Temporal and Causal Relations](https://mlsanity.com/paper/1906.04941/) (2019)
- [Structured Prediction with Partial Labelling through the Infimum Loss](https://mlsanity.com/paper/2003.00920/) (2020)
- [Structured Prediction Theory Based on Factor Graph Complexity](https://mlsanity.com/paper/1605.06443/) (2016)
- [Learning Constraints and Descriptive Segmentation for Subevent Detection](https://mlsanity.com/paper/2109.06316/) (2021)
- [Learnability with Indirect Supervision Signals](https://mlsanity.com/paper/2006.08791/) (2020)
- [Learning Constraints for Structured Prediction Using Rectifier Networks](https://mlsanity.com/paper/2006.01209/) (2020)
- [Train and Test Tightness of LP Relaxations in Structured Prediction](https://mlsanity.com/paper/1511.01419/) (2015)
- [Generalization Bounds for Learning with Linear, Polygonal, Quadratic and Conic Side Knowledge](https://mlsanity.com/paper/1405.7764/) (2014)

## Built on by (most cited first, 3 of 3)

- [On the Independence Assumption in Neurosymbolic Learning](https://mlsanity.com/paper/2404.08458/) (2024)
- [From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verification](https://mlsanity.com/paper/2403.06326/) (2024)
- [Imbalances in Neurosymbolic Learning: Characterization and Mitigating Strategies](https://mlsanity.com/paper/2407.10000/) (2024)

## Abstract (arXiv)

Prior knowledge and symbolic rules in machine learning are often expressed in the form of label constraints, especially in structured prediction problems. In this work, we compare two common strategies for encoding label constraints in a machine learning pipeline, regularization with constraints and constrained inference, by quantifying their impact on model performance. For regularization, we show that it narrows the generalization gap by precluding models that are inconsistent with the constraints. However, its preference for small violations introduces a bias toward a suboptimal model. For constrained inference, we show that it reduces the population risk by correcting a model's violation, and hence turns the violation into an advantage. Given these differences, we further explore the use of two approaches together and propose conditions for constrained inference to compensate for the bias introduced by regularization, aiming to improve both the model complexity and optimal risk.