Proper Calibeating

The classic concept of "calibrated forecasts" and its more recent refinement, "calibeating," are defined with respect to the standard quadratic scoring rule. We extend these notions to the class of proper scoring rules (for which the true distribution is an optimal forecast) and define textitproper calibration and textitproper calibeating by requiring the corresponding guarantees to hold uniformly over all bounded proper scoring rules. We first establish that calibration always implies proper calibration, whereas calibeating need not imply proper calibeating. Second, we show how to guarantee proper calibeating and proper multicalibeating; in particular, textitcomplete calibeating---a strong form of calibeating that calibeats the joint binning---is always proper. Finally, we consider textitdecision-making under uncertainty, where one best replies to the forecasts. We establish that (proper) calibration is equivalent to universal no regret, and (proper) complete calibeating is equivalent to universal no regret together with universal subsuming of the reference sequence, where textituniversal refers to all bounded utility functions.