What is data drift?
Data drift is a change in the statistical distribution of a model’s inputs between training and production. Averages move, ranges widen, categories appear that the model has never seen, and the share of missing values rises or falls. Statisticians call the core case covariate shift. Common sources include a new customer segment, a new sales channel, a replaced sensor, a redesigned web form, and seasons the training data did not cover.
Data drift differs from concept drift, although both are causes of model drift. With data drift the relationship between inputs and outcomes may still hold, but the model now sees more of the cases it learned poorly or never saw. Some data drift has no effect on accuracy. It does offer one practical advantage to the people monitoring a model: it can be detected from the inputs alone, weeks or months before outcomes arrive to confirm a drop in accuracy.
Data drift in practice
Teams detect data drift by comparing each input’s recent distribution with a reference window, usually the training data. The population stability index (PSI), long used in credit scoring, compares the share of records that fall into each value band, and by common convention a PSI above 0.25 signals a major shift. The Kolmogorov-Smirnov test serves numeric inputs, a chi-square test serves categories, and the distribution of the model’s own scores gives a quick summary signal.
A collections agency scores new accounts by likelihood of payment to decide which ones get calls first. The model was trained mostly on medical accounts with mid-sized balances. When a new client places a large portfolio of telecom accounts, the drift report flags balance and account age within the first week, because small balances and recent debt were rare in training. The agency works the portfolio with a separate call strategy while payment outcomes accumulate, then retrains with the new accounts included.
What to watch for
- Weight alerts by how much the model relies on each input, since drift in an unimportant field rarely matters.
- Check the source system before retraining, because a changed unit, a new default value, or a renamed category often looks like drift.
- Choose the reference window deliberately, since a seasonal business will show drift every month against a fixed baseline.
- Run drift checks alongside data observability, because observability catches missing or broken data and drift checks catch data that is valid but different.