Question 1

What's the difference between covariate shift and concept drift?

Accepted Answer

Covariate shift is P(x) moving — inputs changed, the relationship held. Concept drift is P(y|x) moving — the relationship itself changed. The second breaks what the model learned.

Question 2

Why is drift monitoring fundamentally hard?

Accepted Answer

You can see inputs shift immediately, but measuring accuracy needs labels, which arrive months late or never. You're monitoring a proxy that's only loosely coupled to what you care about.

Question 3

What did Rabanser et al. find?

Accepted Answer

Per-feature univariate tests detect the shifts that matter poorly. Reducing dimensionality first — using the model's own representations — then testing works better.

Question 4

When is the drift your own fault?

Accepted Answer

Feedback loops. A recommender shapes what people see, which shapes what they click, which becomes the next training set. No external event to point at.

Question 5

When should you not use Data Drift?

Accepted Answer

(It's a failure mode. The question is when your monitoring lies.)* Input drift as a proxy for accuracy. Inputs can shift with accuracy stable, and vice versa. It's a hint, not a measurement. Significance tests at scale. With enough data everything is significantly different. Use effect size. Per-feature tests alone. They miss the shifts that matter. Test the model's representation. Retraining reflexively on recent data. You may be chasing noise and overfitting to last month.

Question 6

What can you use instead of Data Drift?

Accepted Answer

Labelled sample monitoring — the only real measurement. Buy the labels. Prediction drift — cheap, and it aggregates the input changes that actually mattered. Scheduled retraining — crude, and it's what most production systems do. Shadow deployment — run the new model alongside, compare before switching.

Question 7

What are common mistakes with Data Drift?

Accepted Answer

Treating input drift as evidence of degradation. It's correlated, not equivalent. Firing alerts on statistical significance. At scale, everything is significant. Not noticing the drift is yours. Recommenders shape the data they're retrained on. Assuming a retrain fixes it. You've chosen a window, which encodes an unverifiable assumption about how fast the world moves.

Data Drift

When not to use it

Reach for something else instead

Sources & further reading

Where people go wrong

At a glance

Often compared with

Data Drift

When not to use it

Reach for something else instead

Sources & further reading

Where people go wrong

At a glance

Often compared with

Related concepts

Train/Test Split

Training Data

Recommender System

Time Series Forecasting

A/B Testing

Anomaly Detection