Design ML model monitoring
Hard45 minFree, no account
The model is good on launch day. This is the system that tells you when it stops being.
The question
Design the monitoring that sits around every model your company serves.
Be specific about what you can measure before the labels arrive, which may be weeks.
Functional
- Detect when a model's performance degrades, as early as possible.
- Distinguish a model problem from a data problem from a real change in the world.
- Support safe rollout of a new version, and fast rollback.
Non-functional
- Hundreds of models across many teams.
- Labels arrive anywhere from seconds to sixty days later.
- Alerts must be actionable: a noisy drift alarm gets muted and then ignored.
45:00Commit to an answer before you open the solution. Reading it first teaches you to recognise good answers, which is not the skill being tested.
Stuck?
0 of 3 hints takenThe worked solution
written by a person · not a gradeScore yourself
0 of 5 marked- Layered monitoring by when each signal becomes available25
- Used prediction drift as an early-warning signal20
- Made drift alerts actionable: per feature, importance-weighted, seasonal20
- Distinguished skew from drift and said how to tell20
- A rollout path with shadow, canary and versioned artefacts15
We run no AI here and nothing on this page grades you. The score is yours, and the useful number is the one you get on the same problem a month from now, cold.
kept in this browser only