exchangeability

When production data shifts, how reliable is your conformal coverage?

When production data shifts, your conformal coverage guarantee quietly becomes a historical claim. Split conformal prediction promises that prediction sets will cover the true label at your chosen rate, as long as the calibration set stays exchangeable with future data. But categories get added, ticket wording changes, and a three-month-old calibration set stops being representative. So how do practitioners handle this? Recalibrate on a schedule? Monitor empirical coverage and alert on drops? Or use an adaptive variant?