Understanding Data Leakage

What leakage looks like from the outside

You build a churn model. Cross-validation AUC is 0.91. You deploy it. Real-world AUC is 0.76. Nothing changed — same data, same algorithm, same infrastructure. The model just doesn't work as well as you measured it would.

Data leakage is the most common cause of this gap. Information from outside the training boundary reached the model during development, producing inflated evaluation scores that collapsed the moment they had to generalise to genuinely unseen data.

Leakage is dangerous because it doesn't cause errors. The code runs. The numbers look good. The bug is silent until production.

What leakage actually is

Leakage means that information the model shouldn't have access to at prediction time was made available during training or evaluation. The model learns from that information — so it looks smarter than it is.

There are three patterns:

1. Fit-on-all: A transformer (scaler, imputer, encoder) is fit on the full dataset before the train/test split. The transformer has learned statistics from the test set, so when the test set is evaluated, it's been preprocessed using its own information.

2. Target leakage: A feature is derived from or correlated with the target in a way that wouldn't be available at prediction time. Example: using days_since_cancellation_request as a feature when predicting churn — this column only exists for customers who are already cancelling.

3. Temporal leakage: Data from the future is used to predict the past. Example: computing a customer's 12_month_avg_spend and using it to predict whether they churned in month 3 — the average uses months 4–12, which happened after the prediction point.

Demonstrating fit-on-all leakage quantitatively

The difference between leaked and honest evaluation isn't hypothetical:

import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

df = pd.read_csv("transactions_merged_features.csv")
X = df[["monthly_spend", "support_tickets", "tenure_days", "plan_encoded"]]
y = df["churned"]

# — The leaked version —
scaler = StandardScaler()
X_scaled_all = scaler.fit_transform(X)          # ← fit on ALL data, including test

X_train_l, X_test_l, y_train, y_test = train_test_split(
    X_scaled_all, y, test_size=0.2, random_state=42
)

model = LogisticRegression()
model.fit(X_train_l, y_train)
auc_leaked = roc_auc_score(y_test, model.predict_proba(X_test_l)[:, 1])
print(f"AUC (leaked):  {auc_leaked:.3f}")
# AUC (leaked):  0.912

# — The honest version —
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler_honest = StandardScaler()
X_train_scaled = scaler_honest.fit_transform(X_train)   # ← fit on train only
X_test_scaled  = scaler_honest.transform(X_test)         # ← transform only

model_honest = LogisticRegression()
model_honest.fit(X_train_scaled, y_train)
auc_honest = roc_auc_score(y_test, model_honest.predict_proba(X_test_scaled)[:, 1])
print(f"AUC (honest):  {auc_honest:.3f}")
# AUC (honest):  0.841

Same data, same model, same random seed. AUC drops from 0.912 to 0.841 — a 7-point gap caused entirely by fitting the scaler before splitting. At a business level, this is the difference between a model you'd confidently deploy and one you'd investigate further.

Target leakage: features that know the answer

Target leakage is harder to detect because it's a feature design problem, not a code mistake.

# Feature audit — ask for each column: would I have this at prediction time?
feature_audit = {
    "tenure_days":            "YES — calculated from signup_date, known at any time",
    "monthly_spend":          "YES — last billing cycle is known",
    "support_tickets":        "YES — CRM records exist at prediction time",
    "plan_encoded":           "YES — current plan is known",
    "at_risk_flag":           "DEPENDS — only if the rep note was written before prediction",
    "days_to_first_ticket":   "NO — some customers haven't opened a ticket yet at prediction time",
    "cancellation_flag":      "NO — this only exists for customers already in the churn process",
}

cancellation_flag is a classic target-leakage feature: the model learns to predict churn by noticing that churned customers have a cancellation flag. That's not a prediction — it's a tautology. In production, the flag doesn't exist until the customer is already churning.

The feature audit checklist

For each feature, ask one question: At the moment I make this prediction for a live customer, would this value exist?

If the answer is no — or "sometimes" — the feature needs either to be dropped or redesigned to use only historical data.

Why Pipeline is the structural fix

Every leakage mistake above has the same root cause: manually sequencing fit and transform steps in the wrong order. When you have 5 transformers applied to 3 column groups in a cross-validation loop, keeping track of what's been fit on what is genuinely hard.

sklearn.pipeline.Pipeline makes the correct order structural. You cannot call fit_transform in the wrong sequence because the Pipeline controls the execution. Lesson 2 shows you how to build it.

---

Summary: Data leakage produces inflated evaluation scores that collapse in production. The three patterns are fit-on-all (transformers trained on test data), target leakage (features that encode information about the outcome), and temporal leakage (features derived from future data). Audit every feature against a single question: would this value exist at prediction time? Fit-on-all leakage has a structural fix: sklearn.pipeline.Pipeline, which you'll build in the next lesson.

Next lesson: You know what can go wrong. The next lesson shows you how to make it structurally impossible — by assembling all preprocessing steps into a Pipeline that guarantees the correct fit/transform ordering automatically.