Unit 2 — Model Preparation, Evaluation and Feature Engineering

Topics Covered

Syllabus topics: Data pre-processing; model selection and training (for supervised learning); model representation and interpretability; evaluating machine learning algorithms and performance enhancement of models. What is feature engineering?; feature transformation; feature subset selection; principal component analysis.

2.1 Data pre-processing — the order matters

 1. Split off the test set      ← FIRST. Before anything else touches the data
 2. Handle missing values
 3. Handle outliers
 4. Encode categorical features
 5. Scale numeric features
 6. Balance classes (training set only)

⚠️ Step 1 is first, and this is the single most consequential rule in the unit

Split before you preprocess. If you impute means, scale, or select features using the whole dataset, information from the test set leaks into training, and your test score is no longer an estimate of anything.

The concrete case: you fill missing ages with the mean of all rows. That mean was computed partly from test rows. The model has now seen a summary of data it is supposed to be judged on, and its test score is optimistic — by a little, or by a lot, and you cannot tell which.

The fix is Pipeline, and it is not a stylistic preference:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)      # the scaler learns from TRAIN only
model.score(X_test, y_test)      # and is merely APPLIED to test

Inside a pipeline, fit_transform runs on training data and transform on test data — automatically, and inside every cross-validation fold. Doing it by hand across folds is where leakage actually happens in practice.

⚠️ Which algorithms need scaling — a favourite two-mark question

The rule: if the algorithm measures a distance or penalises coefficient size, scale. If it splits on thresholds, do not bother.

2.2 Model selection and training

   All data (100%)
        │
        ├──────── Training set (60%)   ── fit the model
        ├──────── Validation set (20%) ── choose hyperparameters, compare models
        └──────── Test set (20%)       ── touched ONCE, at the very end

⚠️ Why the test set is touched once

Every time you look at the test score and change something, you fit yourself to the test set. Try twenty models, pick the one with the best test score, and that score is now the maximum of twenty noisy numbers — biased upward, and no longer an estimate of performance on new data.

The validation set exists to absorb that. Report the test score, and report it after you have stopped making decisions.

 fold 1  [ V ][ T ][ T ][ T ][ T ]
 fold 2  [ T ][ V ][ T ][ T ][ T ]
 fold 3  [ T ][ T ][ V ][ T ][ T ]
 fold 4  [ T ][ T ][ T ][ V ][ T ]
 fold 5  [ T ][ T ][ T ][ T ][ V ]
                                     score = mean of the 5

0.9750   0.9625   0.9250   0.9000   0.9500
mean 0.9425      standard deviation 0.0269

⚠️ Never shuffle time series

With time-ordered data, a random split lets the model train on the future and predict the past. The score will be excellent and completely meaningless. Use TimeSeriesSplit, which only ever validates on data after the training fold.

from sklearn.model_selection import GridSearchCV
grid = GridSearchCV(DecisionTreeClassifier(random_state=42),
                    {"max_depth": [1, 2, 3, 5, 10, None]},
                    cv=5, scoring="f1")
grid.fit(X_train, y_train)
grid.best_params_        # chosen on VALIDATION folds, not on test

2.3 The bias–variance trade-off

error
  │╲                                    ╱  total error
  │ ╲                                 ╱
  │  ╲___                          ╱
  │      ╲___              ______╱      variance ──►
  │          ╲____   _____╱
  │  bias ──►      ╳                    ← the sweet spot
  └──────────────────────────────────► model complexity

2.4 Model representation and interpretability

2.5 Evaluating models — and why accuracy usually lies

⚠️ Start here: the base rate

Accuracy alone is nearly meaningless without the base rate.

The churn dataset is 15% positive. Fitting a DummyClassifier that always predicts the majority class:

The dummy is 85% accurate and identifies not one churner. It is worthless, and its accuracy is only 9 points below a real model's.

Always fit a DummyClassifier first. It costs one line, and it converts "94% accuracy" from an impressive-sounding number into a measurable 9-point improvement over doing nothing.

from sklearn.dummy import DummyClassifier
DummyClassifier(strategy="most_frequent").fit(X_train, y_train).score(X_test, y_test)

In fraud detection, where positives are ~0.1%, this argument becomes overwhelming: "never fraud" is 99.9% accurate and catches nothing.

                    PREDICTED
                 Positive   Negative
        Positive    TP         FN        ← actual positives
ACTUAL
        Negative    FP         TN        ← actual negatives

2.6 Performance enhancement

⚠️ Handling imbalance — and where to apply it

Resample the training folds only, never the test set. The test set must keep the real class distribution, or your metrics describe a world that does not exist. Inside cross-validation, that means resampling within each fold — which is another reason to use a Pipeline.

2.7 Feature engineering

⚠️ Target leakage — the most damaging mistake in this course

A feature that would not be available at prediction time, or that encodes the answer.

The symptom is a suspiciously excellent score — 0.99 accuracy, AUC 1.0. If a result looks too good, look for the leak before celebrating. It is almost always there.

The test: would I know this value at the moment I need the prediction? If not, drop it.

2.8 Feature transformation

⚠️ Transformations are fitted, so they leak

A log is a fixed function and is safe anywhere. Anything that learns a parameter from the data — the mean and standard deviation for standardising, the min and max for min–max, the Box–Cox λ, the PCA components — must be fitted on the training set only. That is §2.1's rule, and it is why every one of them belongs inside a Pipeline.

2.9 Feature subset selection

⚠️ Filter methods miss interactions

Two features individually uncorrelated with the target can be strongly predictive together — the classic case being XOR, where each feature alone carries zero information. A filter drops both. A wrapper or an embedded method finds them, and knowing this distinction is the five-mark answer.

⚠️ Selection must happen inside cross-validation

Selecting features on the whole dataset and then cross-validating is leakage, and it produces optimistic scores that will not survive contact with new data. Put the selector in a Pipeline so it is refitted inside every fold.

2.10 Principal Component Analysis

⚠️ PCA's four limitations

1. Components are not interpretable. PC1 is 0.52×sepal length − 0.26×sepal width + 0.58×petal length + 0.56×petal width. That is not a thing anyone can name, and it is why PCA is a poor choice when you must explain the model.

2. It is unsupervised. It maximises variance, not class separation — so a direction with little variance but perfect separating power gets discarded. LDA is the supervised alternative when separation is the goal.

3. It assumes linearity. Data on a curved manifold needs kernel PCA, t-SNE or UMAP.

4. It requires standardisation. Without it, PCA reports whichever feature has the largest units.

Practice problems

Exam questions from this unit

Two marks

1. Why must the test set be split off before preprocessing?
2. Give the formula for precision and for recall.
3. What is the difference between a parameter and a hyperparameter?
4. Which algorithms require feature scaling?
5. What does AUC = 0.5 mean?
6. What is the dummy variable trap?
7. Why does R² never decrease when a predictor is added?

Five marks

1. Explain the data preprocessing pipeline and why order matters.
2. Explain cross-validation and its variants.
3. Distinguish bagging from boosting.
4. Explain the three families of feature selection.
5. Explain data leakage with examples.
6. How would you handle class imbalance?

Ten marks

1. Why is accuracy inadequate? Explain the confusion matrix and derive precision, recall and F1.

2. Explain the bias–variance trade-off, and how to detect and fix overfitting.

3. Explain PCA with a worked example and its limitations.
4. Explain feature engineering and feature selection, with examples of each.

Mistakes that cost marks