Practice Questions with Worked Solutions

A program learns from experience E with respect to tasks T and performance measure P if its performance at T, as measured by P, improves with E.

Classification predicts a category; regression predicts a number.

Classification. It is a linear model fitted like a regression, which is where the name comes from.

2-D, n × p. y is 1-D. A single feature still needs double brackets: df[["hours"]].

Otherwise the transformer learns from test rows, and the test score stops being an estimate of performance on unseen data.

Precision = TP/(TP+FP). Recall = TP/(TP+FN).

No better than random.

Parameters are learned by fitting; hyperparameters are set by you and chosen on validation folds.

Anything measuring a distance or penalising coefficient size — k-NN, K-Means, DBSCAN, SVM, PCA, Ridge/Lasso, neural networks. Trees and Naive Bayes do not.

One-hot encoding k categories into k columns makes them sum to 1 and therefore collinear with the intercept. Drop one for linear and logistic regression.

Least squares can always set the new coefficient to zero, so the fit can only improve or stay the same. Use adjusted R² to compare models of different sizes.

Every feature is assumed conditionally independent of every other, given the class.

The zero-frequency problem — one unseen feature value gives a probability of 0, and since the likelihood is a product, that zeroes the whole posterior.

Fitting only stores the data; all computation happens at prediction time.

A training point lying on the margin. They alone define the hyperplane — remove any other point and the solution is unchanged.

The penalty for margin violations. Large C = narrow margin, less regularised, higher variance. It runs in the opposite direction to alpha.

About 36.8%. The chance of never drawing a row in n draws with replacement is (1 − 1/n)ⁿ → 1/e ≈ 0.368.

WCSS never increases across iterations — but it converges to a local minimum.

The point is closer to a neighbouring cluster than to its own — it is in the wrong cluster.

ARI is corrected for chance, so a random clustering scores 0. The unadjusted index gives random clusterings a score well above zero.

Solution. Draw the contrast:

TRADITIONAL PROGRAMMING              MACHINE LEARNING
   data ──┐                             data ──┐
          ├──► program ──► output              ├──► algorithm ──► MODEL
  rules ──┘                         answers ──┘                     │
                                                  new data ──► prediction

Add when not to use it: known stable rules, very little data, decisions that must be explained, or a simple baseline that already suffices.

Solution. k-fold splits the training data into k parts; each part serves once as validation while the rest train. The score is the mean of the k runs.

One line: bagging averages away variance; boosting chips away at bias.

The distinction that earns the mark: filters miss interactions. Two features individually uncorrelated with the target can be jointly predictive — XOR is the classic case, where each alone carries zero information. A filter drops both.

Add that selection must happen inside cross-validation folds, or it leaks.

Solution. Leakage is information from outside the training data reaching the model, producing a score that cannot be reproduced in production.

1. Target leakage — a feature unavailable at prediction time. total_amount_paid predicting default; discharge_date predicting length of stay.

2. Train–test contamination — imputing or scaling before splitting.

3. Temporal leakage — shuffling time-ordered data so the model trains on the future.

Prevention: split first; put every transformer in a Pipeline; TimeSeriesSplit for time data; and for each feature ask would I know this at the moment I need the prediction?

The symptom is a suspiciously excellent score. AUC 1.0 is nearly always a leak.

Solution. First measure the base rate, because it decides everything: on 15%-positive data a DummyClassifier scores 0.8500 accuracy and 0.0000 recall.

Resample the training folds only, never the test set — the test set must keep the real distribution.

Change the metric too: precision, recall, F1 or AUC-PR, not accuracy.

Open with the base rate and a number. On 15%-positive data, a DummyClassifier predicting the majority class scores accuracy 0.8500 and recall 0.0000 — 85% accurate, and it identifies not one positive case. In fraud detection, where positives are ~0.1%, "never fraud" is 99.9% accurate and catches nothing.

The confusion matrix, naming all four cells, and noting FP is Statistical Foundations for Data Science's Type I error and FN is Type II:

                    PREDICTED
                 Positive   Negative
        Positive    TP         FN
ACTUAL
        Negative    FP         TN

Derive the metrics on 20 countable cases — 6 positive, giving TP 4, FP 4, FN 2, TN 10:

Explain each in words: of those flagged, how many were real versus of the real ones, how many did we catch.

Choose the metric by which error costs more: cancer screening optimises recall (a miss is fatal); a spam filter optimises precision (a real email lost is worse than spam let through).

F1 is the harmonic mean, precisely so a model cannot win by maximising one and abandoning the other — precision 1.0 with recall 0.0 gives F1 = 0, not 0.5.

Finish with AUC. It is threshold-independent and so survives imbalance better: on the worked data, 0.9882 against the dummy's 0.5, where accuracy compressed the same difference into 9 points. On severely imbalanced data prefer the precision–recall curve.

Expected error = bias² + variance + irreducible error. Bias is error from wrong assumptions (too simple); variance is sensitivity to the particular training sample (too complex); irreducible error is noise no model removes.

Draw the U-shaped curve — bias falls and variance rises with complexity.

Detect it:

Then the measured example, which earns the top marks. A decision tree, varying only max_depth:

Training accuracy reaches a perfect 1.0000 while test accuracy falls. A training accuracy of 1.0 is a warning, not an achievement.

Fixes: more data; regularisation (L1/L2); pruning; ensembling — bagging averages variance away; early stopping; fewer features. And cross-validation is how you detect it reliably, because on this data a single split ranges from 0.90 to 0.975.

Definition: an unsupervised technique finding new axes — linear combinations of the original features — ordered by the variance they capture.

The algorithm: standardise; compute the covariance/correlation matrix; find its eigenvalues and eigenvectors; sort by eigenvalue descending; project onto the top k. Eigenvectors are the directions, eigenvalues the variance along them, and the explained variance ratio is eigenvalue ÷ total.

Worked on iris:

The eigenvalues sum to exactly 4 = p — a free check on your arithmetic. Two components carry 95.81%, so four dimensions become two losing 4.19%.

And that 4.19% is literal: the reconstruction MSE at k=2 is 0.0419, exactly 1 minus the variance kept.

How many to keep: the 95% rule says 2; Kaiser (eigenvalue > 1) says 1. They disagree, which is normal — Kaiser under-selects when p is small.

The four limitations:

1. Components are uninterpretable. PC1 is a weighted blend of all four measurements and has no name.

2. It is unsupervised — it maximises variance, not separation. On two dimensions LDA scores 0.9800 against PCA's 0.9133.

3. It assumes linearity. Curved manifolds need kernel PCA, t-SNE or UMAP.

4. It requires standardising. Rescale one iris column to micrometres and PC1 explains 100.0000% — a fact about your units, not your data.

Close with the distinction from feature selection: PCA creates new features and still needs every original at prediction time. If the aim is to stop collecting an expensive measurement, you need selection, not PCA.

State first that logistic regression is a classifier, despite its name.

The sigmoid: p = 1/(1 + e^−z), z = β₀ + β₁x₁ + …, bounded in (0, 1) with p = 0.5 exactly at z = 0. Sketch it.

The interpretation, which carries the marks: odds = p/(1−p); logit = ln(odds) = z, so the model is linear in the log-odds. β is the change in log-odds per unit, and e^β is the odds ratio.

With the worked figure: support_calls has coefficient 2.2656, so e^2.2656 = 9.6367 — one extra standard deviation of calls multiplies the odds of churning by 9.64. Odds are not probability: odds of 2.0 means p = 2/3.

Fitted by maximum likelihood, not least squares, with no closed form.

The threshold is a choice. Show the table:

One model, three different classifiers. Choose the cut-off from the cost of each error.

Finish with multi-class: one-vs-rest, or multinomial softmax.

Bayes' theorem: P(C|X) = P(X|C)P(C)/P(X), naming posterior, likelihood, prior and evidence — and noting the evidence is the same across classes, so it is dropped for comparison.

"Naive" is the conditional independence assumption: P(x₁,…,xₚ|C) = ∏P(xᵢ|C).

Say plainly that it is false, and show it: iris feature correlations reach 0.9629, with three pairs above 0.8 — and GaussianNB still scores 0.9533 CV accuracy. Classification needs only the correct class to rank highest, not the probabilities to be right.

Worked posteriors: P(Yes) × likelihoods = 0.005291, P(No) × likelihoods = 0.020571, normalising to 79.54% No.

The zero-frequency problem: one unseen feature value gives P(xᵢ|C) = 0, and because the likelihood is a product, that single zero drives the posterior to 0.000000 regardless of how strongly the other features support the class.

Laplace smoothing — P(xᵢ|C) = (count + α)/(total + α·k), with α = 1 — restores it to 0.001443.

The variants: Gaussian (continuous), Multinomial (counts — text), Bernoulli (binary). And why it owns spam filtering: linear in the number of features, one training pass, little data needed, updates incrementally.

The measured demonstration. Two interleaved crescents:

And it is not a tuning problem — the best K-Means manages over k = 2…10 is 0.2938. K-Means assigns each point to the nearest centroid, so its boundaries are straight perpendicular bisectors; no k carves out a crescent.

Then DBSCAN's own weakness, honestly. A dense blob and a sparse one, 3 units apart: eps = 0.30 discards the sparse blob as 60 noise points; eps = 1.50 merges them (ARI 0.0178). No single eps recovers both, and K-Means scores 0.8335 here because both blobs are convex.

Neither algorithm is better in general — that is the answer to give.

The difficulty first: with no labels there is no accuracy, and a clustering algorithm always returns clusters — ask K-Means for four groups in pure noise and it gives four, with centroids and a tidy plot.

Internal metrics (geometry only): silhouette ((b−a)/max(a,b); near +1 good, negative means wrong cluster), Davies–Bouldin (low), Calinski–Harabasz (high), and WCSS, which always falls with k and so needs the elbow.

External metrics (labels, for validation only): ARI, corrected for chance; NMI; homogeneity and completeness.

Then the two measured demonstrations that internal metrics are not correctness:

1. On iris, silhouette prefers k = 2 (0.5818) over k = 3 (0.4599) — but there are three species, and k=3's ARI (0.6201) is the higher. Setosa is cleanly separate; versicolor and virginica overlap.

2. Ward linkage agrees best with the truth (ARI 0.6153) and has the worst silhouette (0.4467); single linkage is the reverse.

Conclude honestly: no metric establishes correctness, because correctness is undefined without a purpose. The real tests are whether the clusters are actionable, stable in new data, and nameable by a domain expert.

The algorithm: many decision trees, each on a bootstrap sample, each choosing splits from a random subset of features; majority vote.

The two sources of randomness:

1. Bootstrap sampling — n rows drawn with replacement, so each tree sees about 63.2% of the distinct rows.

2. Random feature subsets at every split — √p features for classification.

Out-of-bag error: each sample omits about 36.8% of rows — because (1 − 1/n)ⁿ → 1/e ≈ 0.368. Predicting each row using only the trees that did not see it gives an unbiased estimate for free.

Then the caution that shows judgement. On iris a 100-tree forest scored 0.8889 on the test split against 0.9778 for a single depth-3 tree — a small clean dataset gives an ensemble nothing to fix, and a 45-row test set is noise. The CV figures (0.9467 ± 0.0267 against 0.9600 ± 0.0249) overlap almost entirely, so the honest conclusion is that the models are not distinguishable here.