How Big Companies Do Model Drift Monitoring

What surprised me most — 6 patterns every company shares

Nobody does drift checks in the request path

Every single company logs inference payloads to a queue (Kafka, Kinesis, Pub/Sub) and runs drift checks async. Zero exceptions. The serving path is sacred — you don't touch it with statistical tests.

Monitor at the feature store, not the model

This blew my mind. Uber and LinkedIn catch drift at the feature store — before it even reaches the model. By the time model outputs drift, you've already been serving bad predictions for a while.

Feature importance weighting is non-negotiable

Raw PSI on every feature is useless noise. Every mature system weights drift severity by feature importance. PSI 0.3 on a throwaway feature? Ignore it. Same score on your top predictor? Wake someone up.

Two-tier compute is how you afford this at scale

Cheap stats (mean, std, null rate) run every minute on everything. Expensive tests (KS, PSI, MMD) only fire when the cheap layer says "something looks off." This cuts compute by 80-90%.

Business metrics are what actually page people

Statistical tests are early warning. But the alert that gets engineers out of bed is CTR dropping, or conversion tanking, or revenue dipping. Drift detection is the canary; business impact is the confirmation.

Almost nobody has fully automated retraining

The dream is L4 (continuous learning). The reality? Most teams are at L2 — scheduled checks, Grafana dashboards, humans making the retrain call. L3 (automated retrain on drift) is the top 10%. L4 is basically mythical.

Company-by-company breakdown

Uber

Michelangelo + Hue observability stack

400+ active models · 15M predictions/sec

Tech stack

Hue — internal observability stack for models
UDQ — Unified Data Quality system
Michelangelo ML platform (since 2016)
Real-time feature health checks
Online/offline feature parity verification

What's unique

Feature health checks run in real time — tracking null rates, detecting drift via statistical tests, and verifying online-offline feature parity. In high-risk cases, model promotions can be blocked entirely until drift issues are resolved. Integrated Gradients (IG) are used for offline batch attribution and regression testing tied to drift detection.

How drift monitoring works at Uber

Michelangelo covers the full ML lifecycle — data management, feature engineering, training, evaluation, deployment, and prediction monitoring. The Hue observability stack tracks both operational metrics (availability, latency, throughput) and prediction-level indicators such as score distributions, calibration, and entropy. Feature values are monitored continuously, and statistical drift tests run to compare production distributions against training baselines. Batch jobs run hourly to monitor prediction drift between ground truth and predictions.

Key insight

Uber learned that monitoring at the feature store level — not at the model level — catches drift before it hits predictions. By the time your model output drifts, you already have degraded user experience. Prevention beats detection.

Extrapolation note

The SHAP importance-weighted PSI scoring described in the overview widget is a reasoned inference. Uber's public posts confirm feature importance tracking and drift detection but do not specify the exact weighting formula used.

Primary sources

ML Model Deployment Safety — Uber Eng Blog 2025 Meet Michelangelo — Uber Eng Blog Integrated Gradients & Drift Detection

Netflix

Metaflow + Runway model management

3,000+ ML projects · petabytes of artifacts

Tech stack

Metaflow — ML workflow orchestration (open-sourced 2019)
Runway — internal model management & drift alerting
Apache Iceberg for feature snapshots
Long-term production performance monitoring
Automated data quality checks

What's unique

Netflix uses an internal tool called Runway to track and notify ML teams of out-of-date models in production. The monitoring tool computes distributions for each attribute and contrasts them against baseline data attributes — from a few days to weeks ago or the actual training data.

How drift monitoring works at Netflix

ML observability at Netflix involves tracking key metrics, detecting anomalies, diagnosing issues, and ensuring models operate reliably. This includes long-term production performance monitoring, feature, target, and prediction drift monitoring, and automated data quality checks. Data scientists can schedule their own notebooks to monitor deployed models. The Runway system alerts teams when model distributions diverge from baselines. SHAP values are used at a group level to explain model decisions — particularly for payment processing, where group-level explanations matter for business conversations with external parties.

Key insight

For recommendation models, business metric shift (engagement, completion rate, CTR) is faster and more reliable than feature drift as a signal. Users notice bad recommendations before any PSI threshold fires. Statistical drift is the early warning; business impact is the confirmation.

Extrapolation note

The "versioned golden dataset snapshots per model release" pattern is inferred from Netflix's versioning practices. Their public posts confirm baseline comparisons exist but do not describe the snapshot lifecycle in detail.

Primary sources

ML Observability — Netflix TechBlog 2025 Supporting Diverse ML Systems — Netflix TechBlog Metaflow — GitHub (Open Source)

Pro-ML / Health Assurance (HA) + Venice + ThirdEye

Billions of daily predictions

Tech stack

Venice — derived data platform / feature store (open-sourced)
Pinot — real-time OLAP for drift statistics
ThirdEye — in-house alerting & monitoring system
InGraphs — real-time feature quantile visualization
Pro-ML Workspace — MLOps portal (launched 2020)

What's unique

LinkedIn's Health Assurance platform tracks drift according to feature importance — not just raw distribution shift. Feature drift graphs are categorized by importance level. Every minute, sample quantiles for each feature are sent to InGraphs for visualization. This is one of the most detailed public descriptions of importance-weighted drift monitoring.

How drift monitoring works at LinkedIn

Feature values computed at inference time are tracked, and a daily job computes statistics on them and pushes them to Pinot. Pinot then pushes these statistics to ThirdEye, which alerts the team whenever a significant change in data distribution is detected. In parallel, every minute, a few sample quantiles for each feature are sent to InGraphs for real-time visualization. The platform lets users choose any two time periods to compare — from training time to the latest inference time. AI engineers are alerted when significant drift is observed in either model output variables or input feature variables.

Key insight (directly from LinkedIn's blog)

LinkedIn's platform reports feature drift categorised by feature importance — a graph shows the timeline of average feature drift across all features, and separately categorises drift by importance rank. This makes triage faster: high-importance feature drift is an incident; low-importance drift is noise.

Open source note

Venice (the feature store powering drift monitoring) has been open-sourced. LinkedIn also open-sourced much of their Frame feature store functionality as the Feathr project.

Primary sources

Model Health Assurance — LinkedIn Eng Blog 2021 One-stop MLOps Portal — LinkedIn Eng Blog 2022 Open Sourcing Venice — LinkedIn Eng Blog 2022

Airbnb

Bighead ML platform + Chronon feature store

Search, pricing, trust models

Tech stack

Bighead — end-to-end ML platform
Chronon — feature engineering platform (open-sourced)
DeepThought — model inference service
Airflow / Redspot — pipeline orchestration
Zipline — feature repository (K/V store)

What's unique

Bighead was purpose-built to solve the problem of model monitoring being done in many places separately — online scores and model features were previously monitored separately, which Bighead unified. Bighead stores model health data as a first-class concept. Chronon handles real-time feature engineering at scale.

How drift monitoring works at Airbnb

Bighead unifies feature engineering, model training, model serving, and monitoring into a single platform. The Bighead Service stores model health data alongside model versions and artifacts. The DeepThought inference service handles model serving, with logging and visualization baked in. Online/offline scoring consistency is a key focus — differences between training-time and inference-time feature values were historically a major source of silent model degradation at Airbnb, and Bighead was explicitly built to eliminate this. Chronon handles real-time aggregation features, processing over 1 million user events per second.

Key insight

Airbnb's core problem statement: models were built in 8–12 weeks with no consistency, online and offline data had significant discrepancies, and monitoring was scattered across notebooks. Bighead was built to solve all three simultaneously. The lesson for any org: unified platforms prevent silent drift more than any monitoring tool.

Extrapolation note

The drift classifier approach (LightGBM with AUC threshold) and correlation matrix monitoring described in the overview widget are reasoned inferences. Airbnb's public posts confirm monitoring and health data storage but do not describe the specific drift detection algorithm used internally.

Primary sources

Bighead Platform — Data Innovation Summit 2019 Bighead Architecture — SlideShare Deck

Tech stack

FBLearner Flow — ML workflow management platform
Scuba — distributed in-memory analytics (stores prediction logs)
Unidash — internal production monitoring dashboard
Dedicated training-data-quality systems
Real-time calibration monitoring & auto-mitigation

What's unique

Meta built dedicated training-data-quality systems that detect label drifts with high accuracy and automatically mitigate abnormal data changes — preventing models from learning from corrupted data. Real-time monitoring and auto-mitigation toolsets guarantee predictions remain well calibrated. The escalation workflow starts with Unidash and dives into Scuba.

How drift monitoring works at Meta

A typical ML engineer at Meta gets paged when an engagement metric (CLICKs, ORDERS, LIKES) trends down — and this most often maps to a model performance issue. The diagnostic workflow: first check Unidash (production monitoring), then dive into Scuba for root cause analysis. Scuba is a distributed in-memory database that stores all prediction logs and makes them available for real-time analytics. For label drift specifically, dedicated data quality systems detect label distribution changes over time with high accuracy. The wide spectrum of ads products requires distinct labeling logics for model training, and data sources for label calculation can be unstable — making continuous label drift detection critical.

Key insight

At Meta's scale, business metric degradation is the primary page trigger, not a statistical test. The statistical systems (Scuba, Unidash) are diagnostic tools used after the business alert fires. This is the most honest description of how ML monitoring works in practice at hyperscale.

Extrapolation note

The two-tier compute architecture (cheap stats every minute → expensive tests on trigger) is a reasoned inference from Meta's scale constraints. Meta's public posts confirm real-time monitoring and auto-mitigation but do not detail the specific tiering of statistical test compute.

Primary sources

ML Prediction Robustness — Engineering at Meta 2024 Introducing FBLearner Flow — Engineering at Meta ML Model Performance Management — Fiddler AI Analysis

Google

TFX (TensorFlow Extended) + TFDV

Petabytes of data validated daily

Tech stack

TFX — TensorFlow Extended (production ML platform)
TFDV — TensorFlow Data Validation (open-sourced)
Apache Beam — data-parallel statistics at scale
Vertex AI Model Monitoring (Cloud offering)
Schema-based validation as data quality gate

What's unique

TFDV is used to analyze and validate petabytes of data at Google every day. Drift detection uses L-infinity distance for categorical features and Jensen-Shannon divergence for numeric features. The schema acts as a contract between the model and incoming data — production data must pass schema validation before reaching the model, catching data quality issues before any statistical test runs.

How drift monitoring works at Google / TFX

TFDV generates a schema from training data and validates every production batch against it — catching not just distributional shift but schema violations such as wrong types, out-of-range values, and new categories. Drift detection is supported between consecutive spans of data (e.g., between span N and span N+1, such as different days of training data). The threshold distances for L-infinity and Jensen-Shannon divergence are configurable, and warnings are triggered when drift exceeds the set threshold. In the production phase, TFDV computes statistics over new data splits, validates against the predefined schema, determines anomalies, and triggers remediation actions — which can include fixing the schema, updating preprocessing steps, or updating the model architecture.

Key insight (directly from Google's documentation)

Google's most impactful contribution to the industry: schema validation as the first line of defence before any statistical test. A feature that changed type or gained a new category will silently break a model — schema validation catches this instantly. TFDV can be used from a notebook environment early in the workflow, well before production deployment, making drift prevention a development-time habit rather than a production-time reaction.

Open source

Both TFX and TFDV are fully open-sourced. TFDV is the most accessible entry point to production-grade drift detection — installable via pip, works on Pandas DataFrames, and requires no TensorFlow models to use.

Primary sources

Introducing TFDV — TensorFlow Blog 2018 TFDV Official Guide — tensorflow.org TFDV Get Started — tensorflow.org Analyzing Data at Scale — Google Cloud Architecture

Maturity levels — where most companies actually are

L1
Reactive
Drift found after user complaints. Manual investigation. No monitoring pipeline at all.
L2
Scheduled checks
Batch PSI/KS jobs run on a schedule. Grafana dashboards exist. Humans decide what action to take.
L3
Automated response
Drift triggers retraining pipeline automatically. Shadow deployment is standard practice. Top 10% of orgs.
L4
Continuous learning
Models retrain on rolling windows. Online updates. Drift prevention, not just detection. Extremely rare.

How big companies actually do drift monitoring in production

Nobody does drift checks in the request path

Monitor at the feature store, not the model

Feature importance weighting is non-negotiable

Two-tier compute is how you afford this at scale

Business metrics are what actually page people

Almost nobody has fully automated retraining

Tech stack

What's unique

How drift monitoring works at Uber

Primary sources

Tech stack

What's unique

How drift monitoring works at Netflix

Primary sources

Tech stack

What's unique

How drift monitoring works at LinkedIn

Primary sources

Tech stack

What's unique

How drift monitoring works at Airbnb

Primary sources

Tech stack

What's unique

How drift monitoring works at Meta

Primary sources

Tech stack

What's unique

How drift monitoring works at Google / TFX

Primary sources