MLOps interview questions

What happens after a model ships: drift, monitoring, deployment and the system design questions around them.

14 questions in this category.

These cards follow a model into production, where ML engineer and system design interviews spend their time: the ways inputs and targets drift and how to measure it, monitoring while the labels are still on their way, training-serving skew, releasing safely with shadow and canary deployments, batch versus online inference and feature stores, recommender staples like cold start and two-tower retrieval, and deciding when to retrain.

What is data drift?

A change in the distribution of a model's inputs between training and production, so the live features stop looking like the data it learned from. When P(X) moves but P(y | X) stays the same, it is called covariate shift.

What is concept drift?

A change in the relationship between the inputs and the target, P(y | X), so the same inputs now call for a different prediction. The model's learned rules go stale even when the input data looks unchanged.

What is model drift?

The decay of a deployed model's performance over time, also called model decay, because the world it scores moves away from the data it learned from. Data drift and concept drift are its usual causes; a broken data pipeline can look the same.

What is training-serving skew?

A mismatch between how features are computed for training and how they are computed when the model serves predictions, so the live model sees inputs that differ from what it learned from. It is a pipeline bug, not a change in the world.

What is the Population Stability Index (PSI)?

A score for how far a variable's distribution has shifted between two samples, usually training and production: bin the variable, then sum (actual% − expected%) × ln(actual% / expected%) over the bins. Below 0.1 is usually read as stable and above 0.25 as a major shift.

How do you monitor a model before its labels arrive?

Watch proxies until the ground truth lands: data quality, drift in the inputs and in the predictions, and, for a well-calibrated classifier, an estimate of performance from its own confidence scores. Confirm with realized performance once the labels arrive.

What is the difference between batch and online inference?

Batch inference scores many rows on a schedule and stores the predictions for later lookup; online inference scores each request on demand, within a latency budget. Batch is cheaper and simpler, while online can use features from the moment of the request.

What is a feature store?

A system that computes, stores and serves machine learning features from one definition: an offline store with the full history for training and a low-latency online store with the latest values for serving. It stops training and serving from drifting apart and lets teams reuse features.

What is shadow deployment?

Running a new model next to the live one on real production traffic, logging its predictions without acting on them. It tests the model on real inputs, latency and load with no effect on users.

What is a canary deployment?

Releasing a new model to a small slice of live traffic, say 1 to 5%, watching errors, latency and business metrics, and widening the rollout in steps while they hold. If they don't, traffic goes back to the old model.

What is the cold start problem?

A recommender or personalization model has nothing to go on for new users or new items that have no interaction history yet. The usual fixes use content features, popularity and onboarding questions until enough interactions arrive.

What is a two-tower model?

A retrieval model with two neural networks, one that embeds the user or query and one that embeds the item, trained so that the dot product of the two embeddings scores relevance. Item embeddings are computed in advance, so millions of candidates can be searched quickly with nearest-neighbour search.

How do you decide when to retrain a model?

Retrain on a schedule when the data changes steadily and retraining is cheap, or on a trigger when measured or estimated performance falls below a threshold. Drift alone is a weak trigger, because not all drift hurts the model.

Why can a model that wins offline lose in an A/B test?

Offline metrics are computed on logged data that the old system shaped, against a proxy target, while the A/B test measures the real outcome on live users. Selection bias in the logs, a weak proxy, feedback loops and added latency all open a gap.