Machine Learning
Intermediate
MLOps Model Monitoring & Drift Detection
Detect statistical feature drift and performance decay in production ML models.
Course Overview
Build continuous monitoring pipelines that alert on feature distribution shifts, concept drift, and prediction latency anomalies in deployed prediction models.
What You Will Learn
Identify data drift, concept drift, and target leakage in live ML pipelines
Calculate statistical drift metrics: Kolmogorov-Smirnov test, PSI, and Wasserstein distance
Build automated data drift monitoring dashboards using Evidently AI
Trigger automated model retraining pipelines upon drift threshold breaches
Monitor prediction latency, memory usage, and input payload validation
Tools & Technologies Used
Evidently AI
Python 3.11
Prometheus
Grafana
FastAPI
Structured Curriculum
2 Modules · 4 Lessons · 200 Minutes Total
Module 1
Module 1: Statistical Drift Metrics & Data Auditing
2 lessons
Module 1: Statistical Drift Metrics & Data Auditing
Measure distribution shifts between training baseline and production data.
-
Data Drift vs. Concept Drift Mechanics
Understand how changing feature distributions corrupt downstream prediction accuracy.
-
Statistical Tests for Drift Identification
Compute PSI (Population Stability Index) and KS-tests on numerical and categorical features.
Module 2
Module 2: Dashboarding, Alerts & Retraining Hooks
2 lessons
Module 2: Dashboarding, Alerts & Retraining Hooks
Integrate real-time monitoring and automated alerts into CI/CD pipelines.
-
Evidently AI Dashboard Integration
Generate visual HTML drift reports and export raw metrics to Prometheus.
-
Automated Retraining Trigger Pipelines
Configure webhook alerts that automatically launch retraining jobs when drift exceeds thresholds.
Practical Project & Capstone Outcome
🚀 Capstone Project
Real-Time Drift Detection & Alerting Service
Build a production monitoring service that inspects incoming inference requests, detects feature drift using Evidently, and posts alert digests to Slack/Webhook endpoints.
Prerequisites
- Python intermediate proficiency
- Basic knowledge of machine learning pipelines
Intended Audience
- MLOps Engineers maintaining production model infrastructure
- Data Scientists responsible for deployed model health and accuracy
Instructor Information
H
Hannah Lindqvist
Course Author & Industry Expert
Hannah Lindqvist is a Principal MLOps Infrastructure Architect who manages model monitoring systems for financial risk predictions.
Frequently Asked Questions
What tools are used for generating monitoring reports?
We use Evidently AI alongside Prometheus and Grafana for metrics visualization.