The Production Gap: From Notebook to Production

A data scientist at a financial services firm trains a churn prediction model in a Jupyter notebook. The notebook is clean, the validation metrics are solid, and the model scores exactly as expected on the holdout set. An AI tool generates deployment boilerplate in twenty minutes. Three weeks later, the model is running in production — and the business intelligence team reports that churn predictions have been uniformly wrong since go-live. No error was raised. The model served predictions continuously. It was just silently computing them against a different distribution than the one it was trained on.

This scenario is not unusual. It is the default outcome when a notebook is promoted to production without understanding the five ways that production environments differ from development environments. This lesson maps those five failure modes. Every subsequent lesson in this course is a deep dive into one of them.

Why notebooks fail in production

A notebook is an excellent environment for exploration. It is a poor environment for production for one fundamental reason: it conflates the development environment with the execution environment. When you run a notebook, the kernel holds your data, your trained objects, your library versions, and your preprocessing state all in memory at once. In production, none of that is true.

The model runs in a separate process, possibly on a different machine, with a different Python environment, receiving data it has never seen from a pipeline it has never touched. Every assumption the notebook made silently — about library versions, input shapes, preprocessing state, and feature availability — is an assumption that production will test.

The five production failure modes

1. Training-serving skew

The most common and most silent failure. The features your model receives at inference time are computed differently from the features it was trained on. The preprocessing code diverges. The data source changes. An aggregation window shifts.

The model does not raise an error. It scores every request. The scores are just wrong, because the model learned relationships in a distribution that no longer matches what it receives.

2. Model staleness

The world changes. Customer behaviour shifts. Regulatory categories update. New product lines alter the input distribution. A model trained in January may be systematically wrong by March — not because of a deployment error, but because the relationship it learned no longer holds.

Without monitoring, you discover this from business outcomes, not from your ML infrastructure.

3. Infrastructure mismatch

The model was trained with scikit-learn 1.2 on Python 3.10. The deployment environment runs scikit-learn 1.0 on Python 3.8. The model loads without error. Predictions differ by up to 15% from expected values because numerical precision and default parameters changed between versions.

This is a packaging problem. It is entirely preventable and entirely invisible without explicit version pinning.

4. Missing monitoring

You cannot detect the first three failures without observability. Most notebooks are shipped to production with no monitoring: no checks on input distributions, no tracking of output distributions, no latency baselines, no alerts.

When something goes wrong, the first signal is a business outcome — a dropped conversion rate, a surge in customer complaints, a compliance audit. By then, the model has been wrong for weeks.

5. Deployment-time regressions

The model passed all tests in the notebook. In production, it is wrapped in a FastAPI service that was AI-generated, containerised, and deployed. The service handles concurrency incorrectly. Under load, prediction latency climbs until the service times out and returns default values — which are logged as predictions.

No error appears in the model's metrics. The service metrics show timeouts, but nobody has configured an alert on them.

Review this code — is it production-ready?

Your team has asked you to deploy a trained model. An AI tool generates the following deployment function:

import joblib

def deploy_model(model, path="model.pkl"):
    joblib.dump(model, path)
    print(f"Model saved to {path}")
    return path

# Usage
trained_model = train_pipeline(X_train, y_train)
deploy_model(trained_model)

Before reading further: what is missing from this function? Identify at least three problems.

---

Here is what is missing:

No version information. The saved file contains no record of the Python version, scikit-learn version, or any other dependency that affects how the model computes predictions. When this file is loaded in a different environment — or in the same environment three months later after a pip upgrade — the predictions may differ silently.

No input schema. The model expects a specific set of features in a specific order. Nothing here stores that contract. When the inference service receives a request, it has no way to validate that the input matches what the model was trained on.

No environment pinning. The function does not capture a requirements.txt, a conda environment, or a Docker image tag. The model artifact is decoupled from its runtime.

These are not edge cases. They are the three most common causes of silent failures in the first week of a model's production life.

What this course teaches

This course is not about writing MLOps pipelines from scratch. It is about developing the judgment to know when a pipeline is safe to ship — and what questions to ask when an AI tool generates one.

By the end of this course, you will be able to:

• Identify training-serving skew in a feature pipeline before it reaches production
• Evaluate a Dockerfile or Kubernetes manifest for production readiness
• Design a monitoring strategy that detects model drift before your users do
• Make informed decisions about retraining triggers and model governance

The next lesson covers model packaging and serialisation — where the first failure mode (infrastructure mismatch) is either prevented or introduced.

Summary

• Notebooks fail in production because they conflate the development and execution environments
• The five production failure modes are: training-serving skew, model staleness, infrastructure mismatch, missing monitoring, and deployment-time regressions
• AI-generated deployment code scaffolds the structure but routinely omits version pinning, schema validation, and monitoring hooks
• Next: Model Packaging and Serialisation — choosing the right format and capturing the runtime contract