Practical Lab

15 experiments, each set out as 1. Question, 2. Aim, 3. Steps, 4. Programme, 5. Execution and Results. Recommended datasets: weather.arff, iris.arff, supermarket.arff, vote.arff, contact-lenses.arff, or custom CSV.

Code lives in labs/course-8-datamining/.

pip install -r tools/data-science/requirements.txt
tools/data-science/setup_weka.sh                  # WEKA and its datasets, in /tmp/weka
python3 tools/data-science/run_data_labs.py       # the Python halves, asserted
bash data-science/labs/course-8-datamining/11_decision_tree_weka.sh   # one WEKA half

WEKA reports how long each model took to build; those lines differ on every run, and capture_lab_outputs.py --check sets them aside.

WEKA in five minutes

Filters are the heart of the Preprocess tab, and they divide two ways:

weka.filters
├── supervised          ← uses the class attribute
│   ├── attribute/      (Discretize, AttributeSelection)
│   └── instance/       (Resample, SMOTE)
└── unsupervised        ← ignores the class
    ├── attribute/      (Normalize, Standardize, Discretize,
    │                    ReplaceMissingValues, Remove, PrincipalComponents,
    │                    NumericToNominal, StringToWordVector)
    └── instance/       (RemoveWithValues, Randomize)

Choosing supervised versus unsupervised Discretize is itself an exam question: the supervised version uses the class label to place cut points where they best separate classes (Fayyad–Irani MDL), and generally produces better bins for a subsequent classifier.

Every Explorer choice is a WEKA class, and the command line names it: the Classify tab's trees/J48 is weka.classifiers.trees.J48, the Preprocess filter unsupervised/attribute/Normalize is weka.filters.unsupervised.attribute.Normalize. That is how the _weka.sh scripts repeat a click-path.

The ARFF format

@relation weather

@attribute outlook     {sunny, overcast, rainy}
@attribute temperature numeric
@attribute humidity    numeric
@attribute windy       {TRUE, FALSE}
@attribute play        {yes, no}

@data
sunny,85,85,FALSE,no
sunny,80,90,TRUE,no
overcast,83,86,FALSE,yes
?,70,96,FALSE,yes          % '?' is a missing value

The last attribute is the class by default. Sparse ARFF, using {index value, index value}, stores only non-zero entries — which is what supermarket.arff uses.

Experiment 1 — Load datasets and explore ARFF/CSV

1. Question

Load datasets in ARFF and CSV formats, and explore their attributes.

2. Aim

Load data into WEKA, read its attribute summary, and convert between ARFF and CSV.

3. Steps

In WEKA, from the command line, 01_load_explore_weka.sh:

1. Load the data, and read the Preprocess panel's summary (Open file...).
2. Load the numeric version, whose temperature and humidity are numbers.
3. Save it as CSV, and load the CSV back (Save..., then Open file... as CSV).
4. Turn a numeric attribute into a nominal one (filter NumericToNominal).

In Python, 01_load_explore.py:

1. Parse the ARFF file, and check its instances and attributes.
2. Describe the data, as WEKA's Preprocess panel does.
3. Write it as CSV and read it back.
4. Turn a numeric-looking code into a category.

4. Programme

In WEKA, from the command line, 01_load_explore_weka.sh:

#!/usr/bin/env bash
# Experiment 1 in WEKA 3.8.7, from the command line: load and explore ARFF and CSV.
# Each command runs the WEKA class that the Explorer runs for the same choice, and prints
# what the Explorer shows in its output pane. The click-path is on the lab page.
# Needs tools/data-science/setup_weka.sh; run as  bash 01_load_explore_weka.sh
set -euo pipefail
source "$(dirname "$0")/weka.sh"

# Step 1: Load the data, and read the Preprocess panel's summary (Open file...)
weka weka.core.Instances $DATA/weather.nominal.arff

# Step 2: Load the numeric version, whose temperature and humidity are numbers
weka weka.core.Instances $DATA/weather.numeric.arff

# Step 3: Save it as CSV, and load the CSV back (Save..., then Open file... as CSV)
weka weka.core.converters.CSVSaver -i $DATA/weather.numeric.arff -o weather.csv
cat weather.csv
weka weka.core.Instances weather.csv

# Step 4: Turn a numeric attribute into a nominal one (filter NumericToNominal)
weka weka.filters.unsupervised.attribute.NumericToNominal -R 2 -i weather.csv -o weather-nominal-temp.arff
weka weka.core.Instances weather-nominal-temp.arff

In Python, 01_load_explore.py:

"""Experiment 1 — Load datasets and explore ARFF/CSV formats.

WEKA equivalent: Preprocess tab -> Open file (see lab.md).
This prints the same summary WEKA's attribute panel shows, so you can compare.
"""
import io
import pandas as pd
from weather import weather_frame, ARFF


def parse_arff(text):
    """A minimal ARFF reader -- enough to show the format's structure.

    Real work uses scipy.io.arff or liac-arff; this exists so the format is
    legible rather than magic.
    """
    attributes, rows, in_data = [], [], False
    for line in text.splitlines():
        line = line.strip()
        if not line or line.startswith("%"):
            continue
        low = line.lower()
        if low.startswith("@attribute"):
            parts = line.split(None, 2)
            name, spec = parts[1], parts[2].strip()
            if spec.startswith("{"):
                kind, domain = "nominal", [v.strip() for v in spec.strip("{}").split(",")]
            else:
                kind, domain = spec.lower(), None
            attributes.append((name, kind, domain))
        elif low.startswith("@data"):
            in_data = True
        elif in_data:
            rows.append([None if v.strip() == "?" else v.strip()
                         for v in line.split(",")])
    return attributes, pd.DataFrame(rows, columns=[a[0] for a in attributes])


def describe(df, name):
    """What WEKA's Preprocess panel reports."""
    print(f"  {name}: {len(df)} instances, {len(df.columns)} attributes")
    for col in df.columns:
        missing = int(df[col].isna().sum())
        if pd.api.types.is_numeric_dtype(df[col]):
            print(f"    {col:14s} numeric  min={df[col].min():g} max={df[col].max():g} "
                  f"mean={df[col].mean():.4f} sd={df[col].std():.4f} missing={missing}")
        else:
            counts = df[col].value_counts().to_dict()
            print(f"    {col:14s} nominal  {counts} missing={missing}")


def main():
    print("Experiment 1 -- Load and explore")

    # Step 1: Parse the ARFF file, and check its instances and attributes
    attributes, df = parse_arff(ARFF)
    assert len(df) == 14, "weather.nominal has 14 instances"
    assert len(attributes) == 5, "and 5 attributes"
    assert all(a[1] == "nominal" for a in attributes), "all five are nominal"
    assert attributes[-1][0] == "play", "the LAST attribute is the class by default"
    # Step 2: Describe the data, as WEKA's Preprocess panel does
    describe(df, "weather.nominal.arff")

    # Round-trip through CSV, which is how WEKA imports non-ARFF data.
    # Step 3: Write it as CSV and read it back
    csv = df.to_csv(index=False)
    back = pd.read_csv(io.StringIO(csv))
    assert back.equals(df), "CSV round-trip must preserve the data"

    # The trap from lab.md: a numeric-looking CATEGORY.
    # Step 4: Turn a numeric-looking code into a category
    df2 = weather_frame()
    df2["ClassID"] = [101, 102, 103] * 4 + [104, 105]
    assert pd.api.types.is_integer_dtype(df2.ClassID), \
        "read as numeric -- every algorithm would treat it as a MAGNITUDE"
    df2["ClassID"] = df2.ClassID.astype("category")
    assert isinstance(df2.ClassID.dtype, pd.CategoricalDtype), \
        "NumericToNominal is the WEKA filter that fixes this"
    print("  numeric-looking category converted (WEKA: NumericToNominal)")
    print("  format checks passed")


if __name__ == "__main__":
    main()

5. Execution and Results

In WEKA, from the command line, 01_load_explore_weka.sh:

$ java weka.core.Instances data/weather.nominal.arff
Relation Name:  weather.symbolic
Num Instances:  14
Num Attributes: 5

     Name                      Type  Nom  Int Real     Missing      Unique  Dist
1 outlook                    Nom 100%   0%   0%     0 /  0%     0 /  0%     3
2 temperature                Nom 100%   0%   0%     0 /  0%     0 /  0%     3
3 humidity                   Nom 100%   0%   0%     0 /  0%     0 /  0%     2
4 windy                      Nom 100%   0%   0%     0 /  0%     0 /  0%     2
5 play                       Nom 100%   0%   0%     0 /  0%     0 /  0%     2


$ java weka.core.Instances data/weather.numeric.arff
Relation Name:  weather
Num Instances:  14
Num Attributes: 5

     Name                      Type  Nom  Int Real     Missing      Unique  Dist
1 outlook                    Nom 100%   0%   0%     0 /  0%     0 /  0%     3
2 temperature                Num   0% 100%   0%     0 /  0%    10 / 71%    12
3 humidity                   Num   0% 100%   0%     0 /  0%     7 / 50%    10
4 windy                      Nom 100%   0%   0%     0 /  0%     0 /  0%     2
5 play                       Nom 100%   0%   0%     0 /  0%     0 /  0%     2


$ java weka.core.converters.CSVSaver -i data/weather.numeric.arff -o weather.csv
outlook,temperature,humidity,windy,play
sunny,85,85,FALSE,no
sunny,80,90,TRUE,no
overcast,83,86,FALSE,yes
rainy,70,96,FALSE,yes
rainy,68,80,FALSE,yes
rainy,65,70,TRUE,no
overcast,64,65,TRUE,yes
sunny,72,95,FALSE,no
sunny,69,70,FALSE,yes
rainy,75,80,FALSE,yes
sunny,75,70,TRUE,yes
overcast,72,90,TRUE,yes
overcast,81,75,FALSE,yes
rainy,71,91,TRUE,no

$ java weka.core.Instances weather.csv
Relation Name:  weather
Num Instances:  14
Num Attributes: 5

     Name                      Type  Nom  Int Real     Missing      Unique  Dist
1 outlook                    Nom 100%   0%   0%     0 /  0%     0 /  0%     3
2 temperature                Num   0% 100%   0%     0 /  0%    10 / 71%    12
3 humidity                   Num   0% 100%   0%     0 /  0%     7 / 50%    10
4 windy                      Nom 100%   0%   0%     0 /  0%     0 /  0%     2
5 play                       Nom 100%   0%   0%     0 /  0%     0 /  0%     2


$ java weka.filters.unsupervised.attribute.NumericToNominal -R 2 -i weather.csv -o weather-nominal-temp.arff

$ java weka.core.Instances weather-nominal-temp.arff
Relation Name:  weather-weka.filters.unsupervised.attribute.NumericToNominal-R2
Num Instances:  14
Num Attributes: 5

     Name                      Type  Nom  Int Real     Missing      Unique  Dist
1 outlook                    Nom 100%   0%   0%     0 /  0%     0 /  0%     3
2 temperature                Nom 100%   0%   0%     0 /  0%    10 / 71%    12
3 humidity                   Num   0% 100%   0%     0 /  0%     7 / 50%    10
4 windy                      Nom 100%   0%   0%     0 /  0%     0 /  0%     2
5 play                       Nom 100%   0%   0%     0 /  0%     0 /  0%     2

In Python, 01_load_explore.py:

Experiment 1 -- Load and explore
  weather.nominal.arff: 14 instances, 5 attributes
    outlook        nominal  {'Sunny': 5, 'Rain': 5, 'Overcast': 4} missing=0
    temperature    nominal  {'Mild': 6, 'Hot': 4, 'Cool': 4} missing=0
    humidity       nominal  {'High': 7, 'Normal': 7} missing=0
    windy          nominal  {'Weak': 8, 'Strong': 6} missing=0
    play           nominal  {'Yes': 9, 'No': 5} missing=0
  numeric-looking category converted (WEKA: NumericToNominal)
  format checks passed

WEKA's summary is the Preprocess panel as a table: Nom, Int and Real are the shares of each attribute's values that are nominal, whole numbers and decimals. Loaded back from CSV, temperature is numeric (Int 100%, 12 distinct values); after NumericToNominal it is nominal, with one label per value. The Python half parses the same ARFF and checks the round trip.

Experiment 2 — Data cleaning and missing values

1. Question

Clean a dataset that has missing values.

2. Aim

Find and replace missing values in WEKA, and see what mean imputation does to the data.

3. Steps

In WEKA, from the command line, 02_missing_values_weka.sh:

1. Load labor.arff, and count its missing values (the Missing column).
2. Replace them, by mean and mode (filter ReplaceMissingValues).
3. Count them again.

In Python, 02_missing_values.py:

1. Impute the worked example's missing ages.
2. See mean imputation shrink the variance.
3. Compare the imputation strategies.
4. Flag the missing values before imputing.
5. Impute after splitting, not before.

4. Programme

In WEKA, from the command line, 02_missing_values_weka.sh:

#!/usr/bin/env bash
# Experiment 2 in WEKA 3.8.7, from the command line: replace missing values.
# Each command runs the WEKA class that the Explorer runs for the same choice, and prints
# what the Explorer shows in its output pane. The click-path is on the lab page.
# Needs tools/data-science/setup_weka.sh; run as  bash 02_missing_values_weka.sh
set -euo pipefail
source "$(dirname "$0")/weka.sh"

# Step 1: Load labor.arff, and count its missing values (the Missing column)
weka weka.core.Instances $DATA/labor.arff

# Step 2: Replace them, by mean and mode (filter ReplaceMissingValues)
weka weka.filters.unsupervised.attribute.ReplaceMissingValues -i $DATA/labor.arff -o labor-filled.arff

# Step 3: Count them again
weka weka.core.Instances labor-filled.arff

In Python, 02_missing_values.py:

"""Experiment 2 — Data cleaning and missing values.

WEKA equivalent: filters/unsupervised/attribute/ReplaceMissingValues.

Demonstrates NUMERICALLY the variance shrinkage warned about in Unit 2 section
2.5 -- the reason mean imputation is more dangerous than it looks.
"""
import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer, KNNImputer


def unit2_worked_example():
    """Section 2.5: ages [25, 30, NULL, 35, 40, NULL, 28]."""
    ages = pd.Series([25, 30, np.nan, 35, 40, np.nan, 28])
    known = ages.dropna()

    assert known.sum() == 158 and len(known) == 5
    assert round(known.mean(), 4) == 31.6
    assert known.median() == 30

    # One outlier moves the mean far more than the median.
    with_outlier = pd.concat([known, pd.Series([200])])
    assert round(with_outlier.mean(), 1) == 59.7
    assert with_outlier.median() == 32.5

    print(f"  2.5: mean {known.mean():.1f}, median {known.median():.0f}; "
          f"add one 200 -> mean {with_outlier.mean():.1f}, "
          f"median {with_outlier.median():.1f}")


def variance_shrinkage():
    """Mean imputation keeps the mean and DESTROYS the spread."""
    rng = np.random.default_rng(42)
    full = rng.normal(50, 10, 1000)

    holed = full.copy()
    holed[rng.choice(1000, 300, replace=False)] = np.nan   # 30% missing

    imputed = SimpleImputer(strategy="mean").fit_transform(holed.reshape(-1, 1)).ravel()

    assert abs(np.nanmean(holed) - imputed.mean()) < 1e-9, "the MEAN is preserved"
    assert imputed.std() < np.nanstd(holed), "but the STANDARD DEVIATION shrinks"

    shrink = 1 - imputed.std() / np.nanstd(holed)
    # Filling 30% of the values with a constant removes their contribution to
    # the spread, so sd falls by roughly 1 - sqrt(0.7) = 16.3%.
    assert 0.10 < shrink < 0.25, shrink

    print(f"  variance shrinkage: sd {np.nanstd(holed):.4f} -> {imputed.std():.4f} "
          f"({shrink:.1%} lost) while the mean is unchanged")
    print(f"       every correlation weakens and every test becomes overconfident")


def imputation_strategies():
    """The methods from Unit 2's table, on data with a deliberate outlier."""
    df = pd.DataFrame({
        "age":    [25, 30, np.nan, 35, 40, np.nan, 28, 200],
        "income": [30000, 45000, 52000, np.nan, 61000, 38000, np.nan, 90000],
        "city":   ["VJA", "GNT", None, "VJA", "VJA", "GNT", None, "HYD"],
    })

    numeric = ["age", "income"]
    mean_i = pd.DataFrame(SimpleImputer(strategy="mean").fit_transform(df[numeric]),
                          columns=numeric)
    med_i = pd.DataFrame(SimpleImputer(strategy="median").fit_transform(df[numeric]),
                         columns=numeric)
    knn_i = pd.DataFrame(KNNImputer(n_neighbors=2).fit_transform(df[numeric]),
                         columns=numeric)
    mode_city = SimpleImputer(strategy="most_frequent").fit_transform(df[["city"]])

    # The 200 drags the mean-imputed value well above the median-imputed one.
    assert mean_i.age[2] > med_i.age[2], "the outlier inflates the mean"
    assert mode_city[2][0] == "VJA", "mode imputation picks the commonest city"
    assert not mean_i.isna().any().any() and not knn_i.isna().any().any()

    print(f"  imputed age: mean {mean_i.age[2]:.2f}, median {med_i.age[2]:.2f}, "
          f"kNN {knn_i.age[2]:.2f}")
    print(f"       the median is unmoved by the 200 -- prefer it on real data")


def missingness_is_information():
    """Section 2.5: add an indicator BEFORE imputing."""
    df = pd.DataFrame({"income": [30000, np.nan, 52000, np.nan, 61000]})
    df["income_missing"] = df.income.isna().astype(int)
    assert df.income_missing.tolist() == [0, 1, 0, 1, 0]
    print("  added an income_missing indicator -- a tree often finds it predictive")


def impute_after_splitting():
    """Fitting the imputer on ALL data leaks test information into training."""
    rng = np.random.default_rng(0)
    train = np.array([10.0, 12.0, np.nan, 11.0])
    test = np.array([1000.0, np.nan])

    correct = SimpleImputer(strategy="mean").fit(train.reshape(-1, 1))
    leaky = SimpleImputer(strategy="mean").fit(
        np.concatenate([train, test]).reshape(-1, 1))

    assert round(float(correct.statistics_[0]), 4) == 11.0
    assert float(leaky.statistics_[0]) > 200, "the test set's 1000 has leaked in"

    print(f"  leakage: fit on train only -> {correct.statistics_[0]:.2f}; "
          f"fit on everything -> {leaky.statistics_[0]:.2f}")


def main():
    print("Experiment 2 -- Missing values")
    # Step 1: Impute the worked example's missing ages
    unit2_worked_example()
    # Step 2: See mean imputation shrink the variance
    variance_shrinkage()
    # Step 3: Compare the imputation strategies
    imputation_strategies()
    # Step 4: Flag the missing values before imputing
    missingness_is_information()
    # Step 5: Impute after splitting, not before
    impute_after_splitting()
    print("  all Unit 2 missing-data claims verified")


if __name__ == "__main__":
    main()

5. Execution and Results

In WEKA, from the command line, 02_missing_values_weka.sh:

$ java weka.core.Instances data/labor.arff
Relation Name:  labor-neg-data
Num Instances:  57
Num Attributes: 17

     Name                      Type  Nom  Int Real     Missing      Unique  Dist
 1 duration                   Num   0%  98%   0%     1 /  2%     0 /  0%     3
 2 wage-increase-first-year   Num   0%  49%  49%     1 /  2%     7 / 12%    17
 3 wage-increase-second-year  Num   0%  47%  33%    11 / 19%     8 / 14%    15
 4 wage-increase-third-year   Num   0%  14%  12%    42 / 74%     6 / 11%     9
 5 cost-of-living-adjustment  Nom  65%   0%   0%    20 / 35%     0 /  0%     3
 6 working-hours              Num   0%  89%   0%     6 / 11%     3 /  5%     8
 7 pension                    Nom  47%   0%   0%    30 / 53%     0 /  0%     3
 8 standby-pay                Num   0%  16%   0%    48 / 84%     6 / 11%     7
 9 shift-differential         Num   0%  54%   0%    26 / 46%     5 /  9%    10
10 education-allowance        Nom  39%   0%   0%    35 / 61%     0 /  0%     2
11 statutory-holidays         Num   0%  93%   0%     4 /  7%     0 /  0%     6
12 vacation                   Nom  89%   0%   0%     6 / 11%     0 /  0%     3
13 longterm-disability-assis  Nom  49%   0%   0%    29 / 51%     0 /  0%     2
14 contribution-to-dental-pl  Nom  65%   0%   0%    20 / 35%     0 /  0%     3
15 bereavement-assistance     Nom  53%   0%   0%    27 / 47%     0 /  0%     2
16 contribution-to-health-pl  Nom  65%   0%   0%    20 / 35%     0 /  0%     3
17 class                      Nom 100%   0%   0%     0 /  0%     0 /  0%     2


$ java weka.filters.unsupervised.attribute.ReplaceMissingValues -i data/labor.arff -o labor-filled.arff

$ java weka.core.Instances labor-filled.arff
Relation Name:  labor-neg-data-weka.filters.unsupervised.attribute.ReplaceMissingValues
Num Instances:  57
Num Attributes: 17

     Name                      Type  Nom  Int Real     Missing      Unique  Dist
 1 duration                   Num   0%  98%   2%     0 /  0%     1 /  2%     4
 2 wage-increase-first-year   Num   0%  49%  51%     0 /  0%     8 / 14%    18
 3 wage-increase-second-year  Num   0%  47%  53%     0 /  0%     8 / 14%    16
 4 wage-increase-third-year   Num   0%  14%  86%     0 /  0%     6 / 11%    10
 5 cost-of-living-adjustment  Nom 100%   0%   0%     0 /  0%     0 /  0%     3
 6 working-hours              Num   0%  89%  11%     0 /  0%     3 /  5%     9
 7 pension                    Nom 100%   0%   0%     0 /  0%     0 /  0%     3
 8 standby-pay                Num   0%  16%  84%     0 /  0%     6 / 11%     8
 9 shift-differential         Num   0%  54%  46%     0 /  0%     5 /  9%    11
10 education-allowance        Nom 100%   0%   0%     0 /  0%     0 /  0%     2
11 statutory-holidays         Num   0%  93%   7%     0 /  0%     0 /  0%     7
12 vacation                   Nom 100%   0%   0%     0 /  0%     0 /  0%     3
13 longterm-disability-assis  Nom 100%   0%   0%     0 /  0%     0 /  0%     2
14 contribution-to-dental-pl  Nom 100%   0%   0%     0 /  0%     0 /  0%     3
15 bereavement-assistance     Nom 100%   0%   0%     0 /  0%     0 /  0%     2
16 contribution-to-health-pl  Nom 100%   0%   0%     0 /  0%     0 /  0%     3
17 class                      Nom 100%   0%   0%     0 /  0%     0 /  0%     2

In Python, 02_missing_values.py:

Experiment 2 -- Missing values
  2.5: mean 31.6, median 30; add one 200 -> mean 59.7, median 32.5
  variance shrinkage: sd 9.7862 -> 8.1877 (16.3% lost) while the mean is unchanged
       every correlation weakens and every test becomes overconfident
  imputed age: mean 59.67, median 32.50, kNN 35.00
       the median is unmoved by the 200 -- prefer it on real data
  added an income_missing indicator -- a tree often finds it predictive
  leakage: fit on train only -> 11.00; fit on everything -> 258.25
  all Unit 2 missing-data claims verified

labor.arff has missing values in 16 of its 17 attributes — 42 of 57 in wage-increase-third-year, 48 in standby-pay. After the filter, every Missing count is 0.

The Python half implements mean, median, mode and k-NN imputation on the same idea and demonstrates the variance shrinkage from Unit 2 §2.5 numerically: imputing 30% of a column with its mean measurably lowers the standard deviation, and the script asserts it.

Experiment 3 — Normalization and discretization

1. Question

Normalise and discretise numeric attributes.

2. Aim

Scale attributes to a common range, and turn numbers into bins three ways.

3. Steps

In WEKA, from the command line, 03_normalize_discretize_weka.sh:

1. Normalise every numeric attribute to [0, 1] (filter Normalize).
2. Discretise into 3 equal-width bins (filter Discretize, bins = 3).
3. And into 3 equal-frequency bins (useEqualFrequency = True).
4. Discretise using the class, by Fayyad-Irani MDL (the supervised Discretize).

In Python, 03_normalize_discretize.py:

1. Normalise by min-max and by z-score.
2. See one outlier crush min-max scaling.
3. Smooth by equal-frequency binning.
4. Discretise the ages into three bins.
5. Binarise a category, rather than numbering it.
6. Find which algorithms need scaling.

4. Programme

In WEKA, from the command line, 03_normalize_discretize_weka.sh:

#!/usr/bin/env bash
# Experiment 3 in WEKA 3.8.7, from the command line: normalise and discretise.
# Each command runs the WEKA class that the Explorer runs for the same choice, and prints
# what the Explorer shows in its output pane. The click-path is on the lab page.
# Needs tools/data-science/setup_weka.sh; run as  bash 03_normalize_discretize_weka.sh
set -euo pipefail
source "$(dirname "$0")/weka.sh"

# Step 1: Normalise every numeric attribute to [0, 1] (filter Normalize)
weka weka.filters.unsupervised.attribute.Normalize -i $DATA/iris.arff -o iris-normalised.arff
weka weka.core.Instances iris-normalised.arff

# Step 2: Discretise into 3 equal-width bins (filter Discretize, bins = 3)
weka weka.filters.unsupervised.attribute.Discretize -B 3 -R first-last -i $DATA/iris.arff -o iris-width.arff
grep "^@attribute" iris-width.arff

# Step 3: And into 3 equal-frequency bins (useEqualFrequency = True)
weka weka.filters.unsupervised.attribute.Discretize -B 3 -F -R first-last -i $DATA/iris.arff -o iris-freq.arff
grep "^@attribute" iris-freq.arff

# Step 4: Discretise using the class, by Fayyad-Irani MDL (the supervised Discretize)
weka weka.filters.supervised.attribute.Discretize -R first-last -c last -i $DATA/iris.arff -o iris-supervised.arff
grep "^@attribute" iris-supervised.arff

In Python, 03_normalize_discretize.py:

"""Experiment 3 — Normalization and discretization.

WEKA equivalents: filters/unsupervised/attribute/{Normalize, Standardize,
Discretize} and filters/supervised/attribute/Discretize.

Reproduces Unit 2 sections 2.9 and 2.10 exactly.
"""
import numpy as np
import pandas as pd
from sklearn.preprocessing import (MinMaxScaler, StandardScaler, RobustScaler,
                                   KBinsDiscretizer)


def unit2_normalisation():
    """Section 2.10: income 73600, min 12000, max 98000, mu 54000, sd 16000."""
    x, lo, hi, mu, sd = 73600, 12000, 98000, 54000, 16000

    assert round((x - lo) / (hi - lo), 4) == 0.7163
    assert round((x - mu) / sd, 4) == 1.225
    j = len(str(hi))                       # smallest j with max|x'| < 1
    assert j == 5 and round(x / 10 ** j, 4) == 0.736

    print(f"  2.10: min-max {(x-lo)/(hi-lo):.4f}, z-score {(x-mu)/sd:.4f}, "
          f"decimal {x/10**j:.4f}")


def outlier_destroys_minmax():
    """Section 2.10: one outlier crushes every other value into a tiny range."""
    v = np.array([[10.0], [12.0], [11.0], [13.0], [1000.0]])
    scaled = MinMaxScaler().fit_transform(v).ravel()

    assert [round(s, 4) for s in scaled] == [0.0, 0.002, 0.001, 0.003, 1.0], scaled
    assert scaled[:4].max() < 0.01, "four sensible values inside 1% of the range"

    z = StandardScaler().fit_transform(v).ravel()
    r = RobustScaler().fit_transform(v).ravel()
    assert abs(z[:4]).max() < abs(z[4]), "z-score also flags the outlier, less brutally"
    assert abs(r[:4]).max() < 2, "robust scaling keeps the sensible values sensible"

    print(f"  outlier: min-max gives {[round(float(s), 4) for s in scaled]}")
    print(f"       robust  gives {[round(float(s), 2) for s in r]}")
    print(f"       min-max crushes the four real values into 0.3% of the range;")
    print(f"       robust keeps them spread and leaves the outlier VISIBLE as one")


def unit2_binning():
    """Section 2.6: prices 4,8,9,15,21,21,24,25,26,28,29,34 in equal-frequency
    bins of depth 4, smoothed three ways."""
    v = [4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29, 34]
    bins = [v[0:4], v[4:8], v[8:12]]

    means = [sum(b) / len(b) for b in bins]
    assert means == [9.0, 22.75, 29.25], means

    medians = [float(np.median(b)) for b in bins]
    assert medians[0] == 8.5

    def boundaries(b):
        lo, hi = min(b), max(b)
        return [lo if abs(x - lo) <= abs(x - hi) else hi for x in b]

    assert boundaries(bins[0]) == [4, 4, 4, 15], boundaries(bins[0])

    print(f"  2.6: bin means {means}; bin-1 boundaries {boundaries(bins[0])}")


def unit2_discretization():
    """Section 2.9: ages 8,15,22,25,31,38,44,51,67 into 3 bins."""
    ages = np.array([8, 15, 22, 25, 31, 38, 44, 51, 67], dtype=float)

    rng = ages.max() - ages.min()
    assert rng == 59
    width = rng / 3
    assert round(width, 2) == 19.67
    edges = [ages.min() + width, ages.min() + 2 * width]
    assert [round(e, 2) for e in edges] == [27.67, 47.33], edges

    ew = KBinsDiscretizer(n_bins=3, encode="ordinal", strategy="uniform",
                          quantile_method="averaged_inverted_cdf"
                          ).fit(ages.reshape(-1, 1))
    got = [round(float(e), 2) for e in ew.bin_edges_[0]]
    assert got == [8.0, 27.67, 47.33, 67.0], got
    counts = np.bincount(ew.transform(ages.reshape(-1, 1)).ravel().astype(int))
    assert counts.tolist() == [4, 3, 2], "equal WIDTH gives uneven counts"

    ef = KBinsDiscretizer(n_bins=3, encode="ordinal", strategy="quantile",
                          quantile_method="averaged_inverted_cdf"
                          ).fit(ages.reshape(-1, 1))
    fcounts = np.bincount(ef.transform(ages.reshape(-1, 1)).ravel().astype(int))
    assert fcounts.tolist() == [3, 3, 3], "equal FREQUENCY gives equal counts"

    print(f"  2.9: equal-width edges {got} -> counts {counts.tolist()}")
    print(f"       equal-frequency -> counts {fcounts.tolist()}")


def binarization_trap():
    """Section 2.9: integer-encoding an UNORDERED category is a bug."""
    colours = ["red", "green", "blue", "red"]

    integer = pd.Series(colours).map({"red": 1, "green": 2, "blue": 3})
    # Under this encoding green is exactly BETWEEN red and blue, which is false.
    assert integer[1] == (integer[0] + integer[2]) / 2, \
        "the arithmetic that integer encoding invents"

    onehot = pd.get_dummies(pd.Series(colours), prefix="is")
    assert list(onehot.columns) == ["is_blue", "is_green", "is_red"]
    assert onehot.sum(axis=1).tolist() == [1, 1, 1, 1], "exactly one 1 per row"

    # The dummy variable trap: k columns are perfectly collinear.
    dropped = pd.get_dummies(pd.Series(colours), drop_first=True)
    assert dropped.shape[1] == 2, "k-1 columns for linear models"

    # An ORDERED category is different -- integers preserve real information.
    ordinal = pd.Series(["low", "high", "medium"]).map({"low": 1, "medium": 2, "high": 3})
    assert ordinal.tolist() == [1, 3, 2]

    print("  one-hot for unordered, integers only where the order is real")
    print("       (k-1 columns for linear models -- the dummy variable trap)")


def which_algorithms_need_scaling():
    """Section 2.10: trees do NOT need scaling; distance methods do."""
    from sklearn.tree import DecisionTreeClassifier
    from sklearn.neighbors import KNeighborsClassifier
    from sklearn.datasets import load_iris
    from sklearn.model_selection import cross_val_score

    X, y = load_iris(return_X_y=True)
    X_skewed = X.copy()
    X_skewed[:, 0] *= 10000                 # blow up one feature's scale

    tree_raw = cross_val_score(DecisionTreeClassifier(random_state=0), X, y, cv=5).mean()
    tree_skew = cross_val_score(DecisionTreeClassifier(random_state=0), X_skewed, y, cv=5).mean()
    assert abs(tree_raw - tree_skew) < 1e-12, \
        "a tree splits on ORDER, which scaling does not change"

    knn_raw = cross_val_score(KNeighborsClassifier(), X, y, cv=5).mean()
    knn_skew = cross_val_score(KNeighborsClassifier(), X_skewed, y, cv=5).mean()
    assert knn_skew < knn_raw, "k-NN is broken by the rescaled feature"

    print(f"  scaling: tree {tree_raw:.4f} -> {tree_skew:.4f} (IDENTICAL); "
          f"k-NN {knn_raw:.4f} -> {knn_skew:.4f} (broken)")


def main():
    print("Experiment 3 -- Normalization and discretization")
    # Step 1: Normalise by min-max and by z-score
    unit2_normalisation()
    # Step 2: See one outlier crush min-max scaling
    outlier_destroys_minmax()
    # Step 3: Smooth by equal-frequency binning
    unit2_binning()
    # Step 4: Discretise the ages into three bins
    unit2_discretization()
    # Step 5: Binarise a category, rather than numbering it
    binarization_trap()
    # Step 6: Find which algorithms need scaling
    which_algorithms_need_scaling()
    print("  all Unit 2 transformation claims verified")


if __name__ == "__main__":
    main()

5. Execution and Results

In WEKA, from the command line, 03_normalize_discretize_weka.sh:

$ java weka.filters.unsupervised.attribute.Normalize -i data/iris.arff -o iris-normalised.arff

$ java weka.core.Instances iris-normalised.arff
Relation Name:  iris-weka.filters.unsupervised.attribute.Normalize-S1.0-T0.0
Num Instances:  150
Num Attributes: 5

     Name                      Type  Nom  Int Real     Missing      Unique  Dist
1 sepallength                Num   0%   1%  99%     0 /  0%     9 /  6%    35
2 sepalwidth                 Num   0%   1%  99%     0 /  0%     5 /  3%    23
3 petallength                Num   0%   1%  99%     0 /  0%    10 /  7%    43
4 petalwidth                 Num   0%   6%  94%     0 /  0%     2 /  1%    22
5 class                      Nom 100%   0%   0%     0 /  0%     0 /  0%     3


$ java weka.filters.unsupervised.attribute.Discretize -B 3 -R first-last -i data/iris.arff -o iris-width.arff
@attribute sepallength {'\'(-inf-5.5]\'','\'(5.5-6.7]\'','\'(6.7-inf)\''}
@attribute sepalwidth {'\'(-inf-2.8]\'','\'(2.8-3.6]\'','\'(3.6-inf)\''}
@attribute petallength {'\'(-inf-2.966667]\'','\'(2.966667-4.933333]\'','\'(4.933333-inf)\''}
@attribute petalwidth {'\'(-inf-0.9]\'','\'(0.9-1.7]\'','\'(1.7-inf)\''}
@attribute class {Iris-setosa,Iris-versicolor,Iris-virginica}

$ java weka.filters.unsupervised.attribute.Discretize -B 3 -F -R first-last -i data/iris.arff -o iris-freq.arff
@attribute sepallength {'\'(-inf-5.45]\'','\'(5.45-6.25]\'','\'(6.25-inf)\''}
@attribute sepalwidth {'\'(-inf-2.85]\'','\'(2.85-3.15]\'','\'(3.15-inf)\''}
@attribute petallength {'\'(-inf-2.45]\'','\'(2.45-4.85]\'','\'(4.85-inf)\''}
@attribute petalwidth {'\'(-inf-0.8]\'','\'(0.8-1.65]\'','\'(1.65-inf)\''}
@attribute class {Iris-setosa,Iris-versicolor,Iris-virginica}

$ java weka.filters.supervised.attribute.Discretize -R first-last -c last -i data/iris.arff -o iris-supervised.arff
@attribute sepallength {'\'(-inf-5.55]\'','\'(5.55-6.15]\'','\'(6.15-inf)\''}
@attribute sepalwidth {'\'(-inf-2.95]\'','\'(2.95-3.35]\'','\'(3.35-inf)\''}
@attribute petallength {'\'(-inf-2.45]\'','\'(2.45-4.75]\'','\'(4.75-inf)\''}
@attribute petalwidth {'\'(-inf-0.8]\'','\'(0.8-1.75]\'','\'(1.75-inf)\''}
@attribute class {Iris-setosa,Iris-versicolor,Iris-virginica}

In Python, 03_normalize_discretize.py:

Experiment 3 -- Normalization and discretization
  2.10: min-max 0.7163, z-score 1.2250, decimal 0.7360
  outlier: min-max gives [0.0, 0.002, 0.001, 0.003, 1.0]
       robust  gives [-1.0, 0.0, -0.5, 0.5, 494.0]
       min-max crushes the four real values into 0.3% of the range;
       robust keeps them spread and leaves the outlier VISIBLE as one
  2.6: bin means [9.0, 22.75, 29.25]; bin-1 boundaries [4, 4, 4, 15]
  2.9: equal-width edges [8.0, 27.67, 47.33, 67.0] -> counts [4, 3, 2]
       equal-frequency -> counts [3, 3, 3]
  one-hot for unordered, integers only where the order is real
       (k-1 columns for linear models -- the dummy variable trap)
  scaling: tree 0.9600 -> 0.9600 (IDENTICAL); k-NN 0.9733 -> 0.7400 (broken)
  all Unit 2 transformation claims verified

The three discretisations of petallength show the difference. Equal width cuts at 2.97 and 4.93, a third of the range each; equal frequency at 2.45 and 4.85, so each bin holds about 50 flowers; the supervised version cuts at 2.45 and 4.75, where the species change.

The Python half reproduces Unit 2's worked examples exactly: min–max of 25 in that twelve-value set is 0.1477; equal-width bins of the ages have edges 27.67 and 47.33; and equal-frequency bins hold three values each. All asserted.

Experiment 4 — Attribute selection and PCA

1. Question

Select the most useful attributes, by a filter and by a wrapper, and reduce the data with PCA.

2. Aim

Rank and select attributes in WEKA, and transform them into principal components.

3. Steps

In WEKA, from the command line, 04_feature_selection_weka.sh:

1. Rank the attributes by information gain (InfoGainAttributeEval, Ranker).
2. Choose a subset with a wrapper round J48 (WrapperSubsetEval, BestFirst).
3. Principal components of the four measurements, with their eigenvalues.
4. Replace the attributes by the components, as the Preprocess filter does.

In Python, 04_feature_selection.py:

1. Rank the attributes by information gain.
2. Select a subset with a wrapper.
3. Read the worked example's eigenvalues.
4. Count the components for 90% of the variance.
5. Standardise before PCA.
6. See what PCA costs in interpretability.

4. Programme

In WEKA, from the command line, 04_feature_selection_weka.sh:

#!/usr/bin/env bash
# Experiment 4 in WEKA 3.8.7, from the command line: attribute selection and PCA.
# Each command runs the WEKA class that the Explorer runs for the same choice, and prints
# what the Explorer shows in its output pane. The click-path is on the lab page.
# Needs tools/data-science/setup_weka.sh; run as  bash 04_feature_selection_weka.sh
set -euo pipefail
source "$(dirname "$0")/weka.sh"

# Step 1: Rank the attributes by information gain (InfoGainAttributeEval, Ranker)
weka weka.attributeSelection.InfoGainAttributeEval -s "weka.attributeSelection.Ranker" -i $DATA/iris.arff

# Step 2: Choose a subset with a wrapper round J48 (WrapperSubsetEval, BestFirst)
weka weka.attributeSelection.WrapperSubsetEval -B weka.classifiers.trees.J48 -F 5 -R 1 -s "weka.attributeSelection.BestFirst" -i $DATA/iris.arff

# Step 3: Principal components of the four measurements, with their eigenvalues
# (PrincipalComponents, Ranker). The class is removed first: given it, this
# evaluator mixes the species into the components as three more numbers.
weka weka.filters.unsupervised.attribute.Remove -R last -i $DATA/iris.arff -o iris-measurements.arff
weka weka.attributeSelection.PrincipalComponents -R 0.95 -s "weka.attributeSelection.Ranker" -i iris-measurements.arff

# Step 4: Replace the attributes by the components, as the Preprocess filter does
# (filters/unsupervised/attribute/PrincipalComponents, varianceCovered = 0.95)
weka weka.filters.unsupervised.attribute.PrincipalComponents -R 0.95 -c last -i $DATA/iris.arff -o iris-pca.arff
grep "^@attribute" iris-pca.arff

In Python, 04_feature_selection.py:

"""Experiment 4 — Attribute selection and PCA.

WEKA: Select attributes tab (InfoGainAttributeEval + Ranker, WrapperSubsetEval
+ BestFirst) and filters/unsupervised/attribute/PrincipalComponents.
"""
import numpy as np
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.feature_selection import mutual_info_classif, RFE
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier


def filter_ranking():
    """WEKA's InfoGainAttributeEval + Ranker."""
    X, y = load_iris(return_X_y=True)
    names = load_iris().feature_names

    mi = mutual_info_classif(X, y, random_state=0)
    ranked = sorted(zip(names, mi), key=lambda p: -p[1])

    # Petal measurements dominate on iris -- the visual separation of Exp 5.
    top_two = {n for n, _ in ranked[:2]}
    assert top_two == {"petal length (cm)", "petal width (cm)"}, top_two

    print("  info gain ranking (WEKA: InfoGainAttributeEval + Ranker)")
    for n, v in ranked:
        print(f"    {v:.4f}  {n}")


def wrapper_selection():
    """WEKA's WrapperSubsetEval: trains the model on each candidate subset."""
    X, y = load_iris(return_X_y=True)
    rfe = RFE(LogisticRegression(max_iter=1000), n_features_to_select=2).fit(X, y)
    kept = [n for n, k in zip(load_iris().feature_names, rfe.support_) if k]
    assert len(kept) == 2

    full = cross_val_score(DecisionTreeClassifier(random_state=0), X, y, cv=5).mean()
    subset = cross_val_score(DecisionTreeClassifier(random_state=0),
                             X[:, rfe.support_], y, cv=5).mean()

    print(f"  wrapper (RFE) kept: {kept}")
    print(f"       4 attributes {full:.4f} vs 2 attributes {subset:.4f}")
    print(f"       half the features, essentially the same accuracy")


def unit2_eigenvalue_table():
    """Section 2.7's worked example: eigenvalues 4.2, 2.1, 0.9, 0.5, 0.3."""
    lam = np.array([4.2, 2.1, 0.9, 0.5, 0.3])
    total = lam.sum()
    assert round(float(total), 4) == 8.0

    prop = lam / total
    cum = np.cumsum(prop)
    assert [round(float(p) * 100, 2) for p in prop[:3]] == [52.50, 26.25, 11.25]
    assert round(float(cum[2]) * 100, 2) == 90.00, "three components give 90%"
    assert int((lam > 1).sum()) == 2, "Kaiser's criterion keeps two"

    print(f"  2.7: three components retain {cum[2]:.1%}; "
          f"Kaiser (lambda>1) keeps {int((lam > 1).sum())}")


def practice_3_eigenvalues():
    """Practice Problem 3: twelve eigenvalues, 90% at five components."""
    lam = np.array([5.4, 2.8, 1.6, 1.1, 0.8, 0.4, 0.3, 0.2, 0.2, 0.1, 0.1, 0.0])
    assert round(float(lam.sum()), 4) == 13.0
    cum = np.cumsum(lam) / lam.sum()

    k90 = int(np.searchsorted(cum, 0.90 - 1e-9) + 1)
    assert k90 == 5, k90
    assert round(float(cum[4]) * 100, 2) == 90.00
    assert int((lam > 1).sum()) == 4, "Kaiser keeps four, retaining 83.85%"
    assert round(float(cum[3]) * 100, 2) == 83.85

    print(f"  Practice 3: 90% at k={k90}; Kaiser gives k=4 at {cum[3]:.2%}")
    print(f"       the two rules DISAGREE, which is normal")


def pca_needs_standardising():
    """Section 2.7: without standardising, PC1 is just the biggest-variance column."""
    X, _ = load_iris(return_X_y=True)
    X_mixed = X.copy()
    X_mixed[:, 0] *= 10000                     # pretend one column is in rupees

    raw = PCA(n_components=2).fit(X_mixed)
    # PC1 is now dominated by that one column.
    assert abs(raw.components_[0][0]) > 0.99, \
        "unstandardised PCA just picks the largest-variance column"
    assert raw.explained_variance_ratio_[0] > 0.999

    std = PCA(n_components=2).fit(StandardScaler().fit_transform(X_mixed))
    assert abs(std.components_[0][0]) < 0.9, "standardised, PC1 blends the columns"

    print(f"  standardising: raw PC1 loading on the inflated column "
          f"{abs(raw.components_[0][0]):.4f} ({raw.explained_variance_ratio_[0]:.4%} "
          f"of variance)")
    print(f"       standardised: {abs(std.components_[0][0]):.4f} -- a real combination")


def pca_costs_interpretability():
    """Section 2.7: components are combinations, so no component is 'age'."""
    X, y = load_iris(return_X_y=True)
    Xs = StandardScaler().fit_transform(X)
    p = PCA(n_components=2).fit(Xs)

    names = load_iris().feature_names
    for i, comp in enumerate(p.components_, 1):
        terms = " ".join(f"{c:+.3f}*{n.split(' ')[0][:5]}" for c, n in zip(comp, names))
        print(f"    PC{i} = {terms}")

    assert all(abs(c).min() > 0.01 for c in p.components_), \
        "EVERY original attribute contributes to every component"
    assert p.explained_variance_ratio_.sum() > 0.95

    print(f"       {p.explained_variance_ratio_.sum():.2%} of variance in 2 of 4 "
          f"dimensions -- but nothing is 'petal length' any more")


def main():
    print("Experiment 4 -- Feature selection and PCA")
    # Step 1: Rank the attributes by information gain
    filter_ranking()
    # Step 2: Select a subset with a wrapper
    wrapper_selection()
    # Step 3: Read the worked example's eigenvalues
    unit2_eigenvalue_table()
    # Step 4: Count the components for 90% of the variance
    practice_3_eigenvalues()
    # Step 5: Standardise before PCA
    pca_needs_standardising()
    # Step 6: See what PCA costs in interpretability
    pca_costs_interpretability()
    print("  all Unit 2 dimensionality claims verified")


if __name__ == "__main__":
    main()

5. Execution and Results

In WEKA, from the command line, 04_feature_selection_weka.sh:

$ java weka.attributeSelection.InfoGainAttributeEval -s weka.attributeSelection.Ranker -i data/iris.arff


=== Attribute Selection on all input data ===

Search Method:
    Attribute ranking.

Attribute Evaluator (supervised, Class (nominal): 5 class):
    Information Gain Ranking Filter

Ranked attributes:
 1.418  3 petallength
 1.378  4 petalwidth
 0.698  1 sepallength
 0.376  2 sepalwidth

Selected attributes: 3,4,1,2 : 4


$ java weka.attributeSelection.WrapperSubsetEval -B weka.classifiers.trees.J48 -F 5 -R 1 -s weka.attributeSelection.BestFirst -i data/iris.arff


=== Attribute Selection on all input data ===

Search Method:
    Best first.
    Start set: no attributes
    Search direction: forward
    Stale search after 5 node expansions
    Total number of subsets evaluated: 11
    Merit of best subset found:    0.947

Attribute Subset Evaluator (supervised, Class (nominal): 5 class):
    Wrapper Subset Evaluator
    Learning scheme: weka.classifiers.trees.J48
    Scheme options: -C 0.25 -M 2
    Subset evaluation: classification accuracy
    Number of folds for accuracy estimation: 5

Selected attributes: 4 : 1
                     petalwidth


$ java weka.filters.unsupervised.attribute.Remove -R last -i data/iris.arff -o iris-measurements.arff

$ java weka.attributeSelection.PrincipalComponents -R 0.95 -s weka.attributeSelection.Ranker -i iris-measurements.arff


=== Attribute Selection on all input data ===

Search Method:
    Attribute ranking.

Attribute Evaluator (unsupervised):
    Principal Components Attribute Transformer

Correlation matrix
  1     -0.11   0.87   0.82
 -0.11   1     -0.42  -0.36
  0.87  -0.42   1      0.96
  0.82  -0.36   0.96   1


eigenvalue  proportion  cumulative
  2.91082     0.7277      0.7277    -0.581petallength-0.566petalwidth-0.522sepallength+0.263sepalwidth
  0.92122     0.23031     0.95801   0.926sepalwidth+0.372sepallength+0.065petalwidth+0.021petallength

Eigenvectors
 V1  V2
-0.5224  0.3723 sepallength
 0.2634  0.9256 sepalwidth
-0.5813  0.0211 petallength
-0.5656  0.0654 petalwidth

Ranked attributes:
 0.2723 1 -0.581petallength-0.566petalwidth-0.522sepallength+0.263sepalwidth
 0.042  2 0.926sepalwidth+0.372sepallength+0.065petalwidth+0.021petallength

Selected attributes: 1,2 : 2


$ java weka.filters.unsupervised.attribute.PrincipalComponents -R 0.95 -c last -i data/iris.arff -o iris-pca.arff
@attribute -0.581petallength-0.566petalwidth-0.522sepallength+0.263sepalwidth numeric
@attribute 0.926sepalwidth+0.372sepallength+0.065petalwidth+0.021petallength numeric
@attribute class {Iris-setosa,Iris-versicolor,Iris-virginica}

In Python, 04_feature_selection.py:

Experiment 4 -- Feature selection and PCA
  info gain ranking (WEKA: InfoGainAttributeEval + Ranker)
    0.9896  petal length (cm)
    0.9749  petal width (cm)
    0.4739  sepal length (cm)
    0.2861  sepal width (cm)
  wrapper (RFE) kept: ['petal length (cm)', 'petal width (cm)']
       4 attributes 0.9600 vs 2 attributes 0.9467
       half the features, essentially the same accuracy
  2.7: three components retain 90.0%; Kaiser (lambda>1) keeps 2
  Practice 3: 90% at k=5; Kaiser gives k=4 at 83.85%
       the two rules DISAGREE, which is normal
  standardising: raw PC1 loading on the inflated column 1.0000 (100.0000% of variance)
       standardised: 0.5211 -- a real combination
    PC1 = +0.521*sepal -0.269*sepal +0.580*petal +0.565*petal
    PC2 = +0.377*sepal +0.923*sepal +0.024*petal +0.067*petal
       95.81% of variance in 2 of 4 dimensions -- but nothing is 'petal length' any more
  all Unit 2 dimensionality claims verified

Information gain ranks petallength (1.418) and petalwidth (1.378) far above the sepal measurements; the wrapper, asking J48 itself, keeps petalwidth alone. Two principal components, with eigenvalues 2.911 and 0.921, cover 95.8% of the variance of the four measurements.

Corrected: this page gave the component's name as 0.348petallength+0.318petalwidth-0.221sepalwidth...; WEKA 3.8.7's is the one above. The script removes the class before the Select-attributes PCA: given it, that evaluator mixes the three species into the components as numbers, which the Preprocess filter does not.

The Python half ranks the iris attributes by information gain, runs PCA, and checks the cumulative variance against Unit 2's worked eigenvalue table.

Experiment 5 — Summarize and visualize

1. Question

Summarise a dataset and visualise it, comparing the classes.

2. Aim

Describe each attribute, and compare the classes on each.

3. Steps

In WEKA, from the command line, 05_summarize_weka.sh:

1. Summarise every attribute (the Preprocess panel).
2. Summarise each class on its own (filter RemoveWithValues, one species at a time).

In Python, 05_summarize.py:

1. Load the iris data.
2. Summarise each attribute.
3. Compare the classes.
4. Find the attribute that separates them.
5. Find the correlations.

4. Programme

In WEKA, from the command line, 05_summarize_weka.sh:

#!/usr/bin/env bash
# Experiment 5 in WEKA 3.8.7, from the command line: summarise the data.
# Each command runs the WEKA class that the Explorer runs for the same choice, and prints
# what the Explorer shows in its output pane. The click-path is on the lab page.
# Needs tools/data-science/setup_weka.sh; run as  bash 05_summarize_weka.sh
set -euo pipefail
source "$(dirname "$0")/weka.sh"

# Step 1: Summarise every attribute (the Preprocess panel)
weka weka.core.Instances $DATA/iris.arff

# Step 2: Summarise each class on its own (filter RemoveWithValues, one species at a time)
for species in 1 2 3; do
  weka weka.filters.unsupervised.instance.RemoveWithValues -C last -L $species -V -i $DATA/iris.arff -o iris-$species.arff
  weka weka.core.Instances iris-$species.arff
done

In Python, 05_summarize.py:

"""Experiment 5 — Summarize and visualize with class-wise comparison.

WEKA: the Preprocess attribute panel, Visualize All, and the Visualize tab's
scatter-plot matrix coloured by class.

This prints the numbers behind those pictures, and asserts the separation that
makes petal length the most informative attribute.
"""
import numpy as np
import pandas as pd
from sklearn.datasets import load_iris


def load():
    d = load_iris(as_frame=True)
    df = d.frame.drop(columns="target")          # the numeric label is not an attribute
    df["species"] = pd.Categorical.from_codes(d.target, d.target_names)
    return df


def attribute_panel(df):
    """What WEKA shows when you click each attribute."""
    print("  attribute summary (WEKA Preprocess panel)")
    for col in df.columns[:-1]:
        s = df[col]
        print(f"    {col:20s} min={s.min():5.2f} max={s.max():5.2f} "
              f"mean={s.mean():6.4f} sd={s.std():6.4f} missing={int(s.isna().sum())}")
    counts = df.species.value_counts().to_dict()
    assert counts == {"setosa": 50, "versicolor": 50, "virginica": 50}, counts
    print(f"    species              {counts}")


def class_wise_comparison(df):
    """The numeric form of 'colour the histogram by class'."""
    stats = df.groupby("species", observed=True).agg(["mean", "std"])
    print("\n  class-wise means")
    for col in df.columns[:-1]:
        means = [f"{stats[(col, 'mean')][s]:.3f}" for s in ["setosa", "versicolor", "virginica"]]
        print(f"    {col:20s} setosa {means[0]}  versicolor {means[1]}  virginica {means[2]}")
    return stats


def separation(df):
    """Petal length separates setosa COMPLETELY -- which is why it has the
    highest information gain in Experiment 4."""
    setosa = df[df.species == "setosa"]["petal length (cm)"]
    others = df[df.species != "setosa"]["petal length (cm)"]

    assert setosa.max() < others.min(), "setosa's petals do not overlap the others at all"
    gap = others.min() - setosa.max()
    assert round(float(gap), 2) == 1.10, gap

    # Sepal width, by contrast, overlaps heavily.
    sw_setosa = df[df.species == "setosa"]["sepal width (cm)"]
    sw_others = df[df.species != "setosa"]["sepal width (cm)"]
    assert sw_setosa.min() < sw_others.max() and sw_others.min() < sw_setosa.max(), \
        "sepal width overlaps -- which is why it ranks last"

    print(f"\n  separation: setosa petal length max {setosa.max():.1f} < "
          f"others min {others.min():.1f} -- a clean gap of {gap:.2f} cm")
    print(f"       a single threshold at 2.5 cm classifies setosa perfectly")

    threshold = 2.45
    predicted_setosa = df["petal length (cm)"] < threshold
    assert (predicted_setosa == (df.species == "setosa")).all(), \
        "one rule, 100% accurate for setosa"


def correlations(df):
    c = df[df.columns[:-1]].corr()
    petal = c.loc["petal length (cm)", "petal width (cm)"]
    assert petal > 0.95, "petal length and width are nearly redundant"
    print(f"\n  correlation: petal length vs width = {petal:.4f}")
    print(f"       nearly redundant -- feature SELECTION would drop one")


def main():
    print("Experiment 5 -- Summarize and visualize")
    # Step 1: Load the iris data
    df = load()
    # Step 2: Summarise each attribute
    attribute_panel(df)
    # Step 3: Compare the classes
    class_wise_comparison(df)
    # Step 4: Find the attribute that separates them
    separation(df)
    # Step 5: Find the correlations
    correlations(df)
    print("\n  summary statistics verified")


if __name__ == "__main__":
    main()

5. Execution and Results

In WEKA, from the command line, 05_summarize_weka.sh:

$ java weka.core.Instances data/iris.arff
Relation Name:  iris
Num Instances:  150
Num Attributes: 5

     Name                      Type  Nom  Int Real     Missing      Unique  Dist
1 sepallength                Num   0%  11%  89%     0 /  0%     9 /  6%    35
2 sepalwidth                 Num   0%  19%  81%     0 /  0%     5 /  3%    23
3 petallength                Num   0%   9%  91%     0 /  0%    10 /  7%    43
4 petalwidth                 Num   0%   9%  91%     0 /  0%     2 /  1%    22
5 class                      Nom 100%   0%   0%     0 /  0%     0 /  0%     3


$ java weka.filters.unsupervised.instance.RemoveWithValues -C last -L 1 -V -i data/iris.arff -o iris-1.arff

$ java weka.core.Instances iris-1.arff
Relation Name:  iris-weka.filters.unsupervised.instance.RemoveWithValues-S0.0-Clast-L1-V
Num Instances:  50
Num Attributes: 5

     Name                      Type  Nom  Int Real     Missing      Unique  Dist
1 sepallength                Num   0%  16%  84%     0 /  0%     4 /  8%    15
2 sepalwidth                 Num   0%  14%  86%     0 /  0%     6 / 12%    16
3 petallength                Num   0%   2%  98%     0 /  0%     2 /  4%     9
4 petalwidth                 Num   0%   0% 100%     0 /  0%     2 /  4%     6
5 class                      Nom 100%   0%   0%     0 /  0%     0 /  0%     1


$ java weka.filters.unsupervised.instance.RemoveWithValues -C last -L 2 -V -i data/iris.arff -o iris-2.arff

$ java weka.core.Instances iris-2.arff
Relation Name:  iris-weka.filters.unsupervised.instance.RemoveWithValues-S0.0-Clast-L2-V
Num Instances:  50
Num Attributes: 5

     Name                      Type  Nom  Int Real     Missing      Unique  Dist
1 sepallength                Num   0%  14%  86%     0 /  0%     8 / 16%    21
2 sepalwidth                 Num   0%  18%  82%     0 /  0%     3 /  6%    14
3 petallength                Num   0%  14%  86%     0 /  0%     6 / 12%    19
4 petalwidth                 Num   0%  14%  86%     0 /  0%     2 /  4%     9
5 class                      Nom 100%   0%   0%     0 /  0%     0 /  0%     1


$ java weka.filters.unsupervised.instance.RemoveWithValues -C last -L 3 -V -i data/iris.arff -o iris-3.arff

$ java weka.core.Instances iris-3.arff
Relation Name:  iris-weka.filters.unsupervised.instance.RemoveWithValues-S0.0-Clast-L3-V
Num Instances:  50
Num Attributes: 5

     Name                      Type  Nom  Int Real     Missing      Unique  Dist
1 sepallength                Num   0%   4%  96%     0 /  0%     9 / 18%    21
2 sepalwidth                 Num   0%  24%  76%     0 /  0%     2 /  4%    13
3 petallength                Num   0%  10%  90%     0 /  0%     5 / 10%    20
4 petalwidth                 Num   0%  12%  88%     0 /  0%     3 /  6%    12
5 class                      Nom 100%   0%   0%     0 /  0%     0 /  0%     1

In Python, 05_summarize.py:

Experiment 5 -- Summarize and visualize
  attribute summary (WEKA Preprocess panel)
    sepal length (cm)    min= 4.30 max= 7.90 mean=5.8433 sd=0.8281 missing=0
    sepal width (cm)     min= 2.00 max= 4.40 mean=3.0573 sd=0.4359 missing=0
    petal length (cm)    min= 1.00 max= 6.90 mean=3.7580 sd=1.7653 missing=0
    petal width (cm)     min= 0.10 max= 2.50 mean=1.1993 sd=0.7622 missing=0
    species              {'setosa': 50, 'versicolor': 50, 'virginica': 50}

  class-wise means
    sepal length (cm)    setosa 5.006  versicolor 5.936  virginica 6.588
    sepal width (cm)     setosa 3.428  versicolor 2.770  virginica 2.974
    petal length (cm)    setosa 1.462  versicolor 4.260  virginica 5.552
    petal width (cm)     setosa 0.246  versicolor 1.326  virginica 2.026

  separation: setosa petal length max 1.9 < others min 3.0 -- a clean gap of 1.10 cm
       a single threshold at 2.5 cm classifies setosa perfectly

  correlation: petal length vs width = 0.9629
       nearly redundant -- feature SELECTION would drop one

  summary statistics verified

The plots are the Explorer's to draw; the command line gives their numbers. One summary per species shows how far apart they are: setosa's petal lengths take only 9 distinct values, all small.

The Python half prints per-class means and standard deviations and confirms the separation numerically.

Experiment 6 — Association rules with Apriori

1. Question

Mine association rules from transaction data with Apriori.

2. Aim

Find frequent itemsets and strong rules with Apriori, and read support, confidence and lift.

3. Steps

In WEKA, from the command line, 06_apriori_weka.sh:

1. Mine the supermarket data with Apriori, for the 10 best rules (Associate tab).
2. The same data, ranked by lift instead of confidence (metricType = Lift).

In Python, 06_apriori.py:

1. Trace Apriori on the five transactions.
2. Mine the nine transactions of Practice Problem 1.
3. See where confidence misleads, and lift does not.
4. Find the negative association in tea and coffee.

4. Programme

In WEKA, from the command line, 06_apriori_weka.sh:

#!/usr/bin/env bash
# Experiment 6 in WEKA 3.8.7, from the command line: association rules with Apriori.
# Each command runs the WEKA class that the Explorer runs for the same choice, and prints
# what the Explorer shows in its output pane. The click-path is on the lab page.
# Needs tools/data-science/setup_weka.sh; run as  bash 06_apriori_weka.sh
set -euo pipefail
source "$(dirname "$0")/weka.sh"

# Step 1: Mine the supermarket data with Apriori, for the 10 best rules (Associate tab)
weka weka.associations.Apriori -N 10 -T 0 -C 0.9 -D 0.05 -U 1.0 -M 0.1 -t $DATA/supermarket.arff

# Step 2: The same data, ranked by lift instead of confidence (metricType = Lift)
weka weka.associations.Apriori -N 10 -T 1 -C 1.1 -D 0.05 -U 1.0 -M 0.1 -t $DATA/supermarket.arff

In Python, 06_apriori.py:

"""Experiment 6 — Association rules with Apriori.

WEKA equivalent: Associate tab -> Apriori (see lab.md).

This script reproduces the hand-traced examples from Unit 3 EXACTLY, so the
notes and the code check each other. Every assertion below corresponds to a
number written out in notes/sem-4/course-8-data-mining/unit-3.md.

Run:  python3 tools/run_data_labs.py     (or this file directly)
"""
import pandas as pd
from mlxtend.frequent_patterns import apriori, association_rules
from mlxtend.preprocessing import TransactionEncoder


def frequent_itemsets(transactions, min_support):
    """One-hot encode the baskets and run Apriori."""
    te = TransactionEncoder()
    frame = pd.DataFrame(te.fit(transactions).transform(transactions),
                         columns=te.columns_)
    return apriori(frame, min_support=min_support, use_colnames=True)


def as_dict(itemsets, n):
    """{frozenset -> support COUNT}, which is what the notes tabulate."""
    return {frozenset(r.itemsets): round(r.support * n) for r in itemsets.itertuples()}


def unit3_main_trace():
    """Unit 3 section 3.4: five transactions, minsup count 3 (60%)."""
    T = [list("ACD"), list("BCE"), list("ABCE"), list("BE"), list("ABCE")]
    n = len(T)

    items = frequent_itemsets(T, 3 / n)
    found = as_dict(items, n)

    expected = {
        frozenset("A"): 3, frozenset("B"): 4, frozenset("C"): 4, frozenset("E"): 4,
        frozenset("AC"): 3, frozenset("BC"): 3, frozenset("BE"): 4, frozenset("CE"): 3,
        frozenset("BCE"): 3,
    }
    assert found == expected, f"\n got      {found}\n expected {expected}"
    assert len(found) == 9, "the notes claim nine frequent itemsets"
    assert frozenset("D") not in found, "{D} has support 1 and must be pruned"

    # Rule generation from {B,C,E} with minconf 0.8
    rules = association_rules(items, metric="confidence", min_threshold=0.8)
    strong = {
        (frozenset(r.antecedents), frozenset(r.consequents)): (round(r.confidence, 4),
                                                               round(r.lift, 4))
        for r in rules.itertuples()
        if frozenset(r.antecedents) | frozenset(r.consequents) == frozenset("BCE")
    }
    assert strong == {
        (frozenset("BC"), frozenset("E")): (1.0, 1.25),
        (frozenset("CE"), frozenset("B")): (1.0, 1.25),
    }, strong

    print(f"  3.4 trace: {len(found)} frequent itemsets, "
          f"{len(strong)} strong rules from {{B,C,E}} -- matches the notes")
    return found


def unit3_practice_1():
    """Unit 3 Practice Problem 1: nine transactions, minsup count 2, minconf 0.7."""
    T = [list("ABE"), list("BD"), list("BC"), list("ABD"), list("AC"),
         list("BC"), list("AC"), list("ABCE"), list("ABC")]
    n = len(T)

    items = frequent_itemsets(T, 2 / n)
    found = as_dict(items, n)

    expected = {
        frozenset("A"): 6, frozenset("B"): 7, frozenset("C"): 6,
        frozenset("D"): 2, frozenset("E"): 2,
        frozenset("AB"): 4, frozenset("AC"): 4, frozenset("AE"): 2,
        frozenset("BC"): 4, frozenset("BD"): 2, frozenset("BE"): 2,
        frozenset("ABC"): 2, frozenset("ABE"): 2,
    }
    assert found == expected, f"\n got      {found}\n expected {expected}"
    assert len(found) == 13, "the notes claim thirteen frequent itemsets"

    # {A,D}, {C,D}, {C,E}, {D,E} are infrequent, so every superset is pruned.
    for gone in ("AD", "CD", "CE", "DE", "ABD", "ACE", "BCE", "BCD", "BDE", "ABCE"):
        assert frozenset(gone) not in found, f"{{{gone}}} must not be frequent"

    rules = association_rules(items, metric="confidence", min_threshold=0.7)

    def summarise(pred):
        return sorted((sorted(r.antecedents), sorted(r.consequents),
                       round(r.confidence, 4), round(r.lift, 4))
                      for r in rules.itertuples() if pred(r))

    # The question asks for rules from the LARGEST frequent itemsets, which is
    # what the notes tabulate: the two 3-itemsets.
    from_3 = summarise(lambda r: len(frozenset(r.antecedents) | frozenset(r.consequents)) == 3)
    assert from_3 == [
        (["A", "E"], ["B"], 1.0, 1.2857),
        (["B", "E"], ["A"], 1.0, 1.5),
        (["E"], ["A", "B"], 1.0, 2.25),
    ], from_3

    # No strong rule comes from {A,B,C}: every confidence there is at most 0.5.
    from_abc = [r for r in rules.itertuples()
                if frozenset(r.antecedents) | frozenset(r.consequents) == frozenset("ABC")]
    assert from_abc == [], "the notes claim {A,B,C} yields no strong rule"

    # Mining the WHOLE itemset lattice also yields three strong rules from
    # 2-itemsets. The exam question scoped itself to the largest itemsets, but
    # a real Apriori run reports these too -- worth knowing they exist.
    from_2 = summarise(lambda r: len(frozenset(r.antecedents) | frozenset(r.consequents)) == 2)
    assert from_2 == [
        (["D"], ["B"], 1.0, 1.2857),
        (["E"], ["A"], 1.0, 1.5),
        (["E"], ["B"], 1.0, 1.2857),
    ], from_2

    print(f"  Practice 1: {len(found)} frequent itemsets, {len(from_3)} strong "
          f"rules from the 3-itemsets (+{len(from_2)} from 2-itemsets)")


def confidence_is_misleading():
    """Unit 3 section 3.2: the cricket / data science example.

    Support 0.72 and confidence 0.90 look excellent; lift is exactly 1.00,
    meaning the two are INDEPENDENT and the rule is worthless.
    """
    n = 1000
    T = ([list("cd")] * 720 + [list("c")] * 80
         + [list("d")] * 180 + [list("x")] * 20)
    assert len(T) == n

    items = frequent_itemsets(T, 0.1)
    rules = association_rules(items, metric="confidence", min_threshold=0.5)
    r = next(x for x in rules.itertuples()
             if frozenset(x.antecedents) == {"c"} and frozenset(x.consequents) == {"d"})

    assert round(r.support, 4) == 0.72
    assert round(r.confidence, 4) == 0.90
    assert round(r.lift, 4) == 1.0, "lift must be exactly 1 -- independent"

    print(f"  3.2 trap: support {r.support:.2f}, confidence {r.confidence:.2f}, "
          f"lift {r.lift:.2f} -- high confidence, ZERO interest")


def tea_and_coffee():
    """Unit 3 Practice Problem 3: a NEGATIVE association hiding behind
    respectable support and confidence."""
    both, tea_only, coffee_only = 400, 200, 350
    neither = 1000 - both - tea_only - coffee_only
    assert neither == 50

    T = ([["tea", "coffee"]] * both + [["tea"]] * tea_only
         + [["coffee"]] * coffee_only + [["water"]] * neither)

    items = frequent_itemsets(T, 0.05)
    rules = association_rules(items, metric="support", min_threshold=0.05)
    r = next(x for x in rules.itertuples()
             if frozenset(x.antecedents) == {"tea"} and frozenset(x.consequents) == {"coffee"})

    assert round(r.support, 4) == 0.40
    assert round(r.confidence, 4) == 0.6667
    assert round(r.lift, 4) == 0.8889, "lift below 1 -- a NEGATIVE association"
    assert round(r.leverage, 4) == -0.05, "leverage agrees: negative"

    print(f"  Practice 3: confidence {r.confidence:.4f} looks fine, but lift "
          f"{r.lift:.4f} < 1 -- tea drinkers are LESS likely to drink coffee")


def main():
    print("Experiment 6 -- Apriori (mlxtend)")
    # Step 1: Trace Apriori on the five transactions
    unit3_main_trace()
    # Step 2: Mine the nine transactions of Practice Problem 1
    unit3_practice_1()
    # Step 3: See where confidence misleads, and lift does not
    confidence_is_misleading()
    # Step 4: Find the negative association in tea and coffee
    tea_and_coffee()
    print("  all Unit 3 hand calculations reproduced")


if __name__ == "__main__":
    main()

5. Execution and Results

In WEKA, from the command line, 06_apriori_weka.sh:

$ java weka.associations.Apriori -N 10 -T 0 -C 0.9 -D 0.05 -U 1.0 -M 0.1 -t data/supermarket.arff

Apriori
=======

Minimum support: 0.15 (694 instances)
Minimum metric <confidence>: 0.9
Number of cycles performed: 17

Generated sets of large itemsets:

Size of set of large itemsets L(1): 44

Size of set of large itemsets L(2): 380

Size of set of large itemsets L(3): 910

Size of set of large itemsets L(4): 633

Size of set of large itemsets L(5): 105

Size of set of large itemsets L(6): 1

Best rules found:

 1. biscuits=t frozen foods=t fruit=t total=high 788 ==> bread and cake=t 723    <conf:(0.92)> lift:(1.27) lev:(0.03) [155] conv:(3.35)
 2. baking needs=t biscuits=t fruit=t total=high 760 ==> bread and cake=t 696    <conf:(0.92)> lift:(1.27) lev:(0.03) [149] conv:(3.28)
 3. baking needs=t frozen foods=t fruit=t total=high 770 ==> bread and cake=t 705    <conf:(0.92)> lift:(1.27) lev:(0.03) [150] conv:(3.27)
 4. biscuits=t fruit=t vegetables=t total=high 815 ==> bread and cake=t 746    <conf:(0.92)> lift:(1.27) lev:(0.03) [159] conv:(3.26)
 5. party snack foods=t fruit=t total=high 854 ==> bread and cake=t 779    <conf:(0.91)> lift:(1.27) lev:(0.04) [164] conv:(3.15)
 6. biscuits=t frozen foods=t vegetables=t total=high 797 ==> bread and cake=t 725    <conf:(0.91)> lift:(1.26) lev:(0.03) [151] conv:(3.06)
 7. baking needs=t biscuits=t vegetables=t total=high 772 ==> bread and cake=t 701    <conf:(0.91)> lift:(1.26) lev:(0.03) [145] conv:(3.01)
 8. biscuits=t fruit=t total=high 954 ==> bread and cake=t 866    <conf:(0.91)> lift:(1.26) lev:(0.04) [179] conv:(3)
 9. frozen foods=t fruit=t vegetables=t total=high 834 ==> bread and cake=t 757    <conf:(0.91)> lift:(1.26) lev:(0.03) [156] conv:(3)
10. frozen foods=t fruit=t total=high 969 ==> bread and cake=t 877    <conf:(0.91)> lift:(1.26) lev:(0.04) [179] conv:(2.92)

=== Evaluation ===

Elapsed time: 2.441s


$ java weka.associations.Apriori -N 10 -T 1 -C 1.1 -D 0.05 -U 1.0 -M 0.1 -t data/supermarket.arff

Apriori
=======

Minimum support: 0.35 (1619 instances)
Minimum metric <lift>: 1.1
Number of cycles performed: 13

Generated sets of large itemsets:

Size of set of large itemsets L(1): 22

Size of set of large itemsets L(2): 36

Size of set of large itemsets L(3): 3

Best rules found:

 1. fruit=t 2962 ==> bread and cake=t vegetables=t 1791    conf:(0.6) < lift:(1.22)> lev:(0.07) [319] conv:(1.27)
 2. bread and cake=t vegetables=t 2298 ==> fruit=t 1791    conf:(0.78) < lift:(1.22)> lev:(0.07) [319] conv:(1.63)
 3. vegetables=t 2961 ==> bread and cake=t fruit=t 1791    conf:(0.6) < lift:(1.2)> lev:(0.07) [303] conv:(1.26)
 4. bread and cake=t fruit=t 2325 ==> vegetables=t 1791    conf:(0.77) < lift:(1.2)> lev:(0.07) [303] conv:(1.56)
 5. baking needs=t 2795 ==> margarine=t 1645    conf:(0.59) < lift:(1.19)> lev:(0.06) [262] conv:(1.23)
 6. margarine=t 2288 ==> baking needs=t 1645    conf:(0.72) < lift:(1.19)> lev:(0.06) [262] conv:(1.41)
 7. frozen foods=t 2717 ==> biscuits=t 1810    conf:(0.67) < lift:(1.18)> lev:(0.06) [280] conv:(1.31)
 8. biscuits=t 2605 ==> frozen foods=t 1810    conf:(0.69) < lift:(1.18)> lev:(0.06) [280] conv:(1.35)
 9. vegetables=t 2961 ==> fruit=t 2207    conf:(0.75) < lift:(1.16)> lev:(0.07) [311] conv:(1.41)
10. fruit=t 2962 ==> vegetables=t 2207    conf:(0.75) < lift:(1.16)> lev:(0.07) [311] conv:(1.41)

=== Evaluation ===

Elapsed time: 0.744s

In Python, 06_apriori.py:

Experiment 6 -- Apriori (mlxtend)
  3.4 trace: 9 frequent itemsets, 2 strong rules from {B,C,E} -- matches the notes
  Practice 1: 13 frequent itemsets, 3 strong rules from the 3-itemsets (+3 from 2-itemsets)
  3.2 trap: support 0.72, confidence 0.90, lift 1.00 -- high confidence, ZERO interest
  Practice 3: confidence 0.6667 looks fine, but lift 0.8889 < 1 -- tea drinkers are LESS likely to drink coffee
  all Unit 3 hand calculations reproduced

Every one of the ten best rules by confidence ends in bread and cake: with confidence 0.91–0.92 and lift 1.26–1.27, a large basket that holds biscuits and fruit almost always holds bread too. Ranked by lift instead, the rules change — fruit with bread and cake and vegetables comes first, at lift 1.22 but confidence only 0.6.

The Python half uses mlxtend and reproduces Unit 3 §3.4's trace exactly — the same five transactions, minsup 0.6, giving the nine frequent itemsets and the two strong rules {B,C}→{E} and {C,E}→{B}, each with confidence 1.00 and lift 1.25. It also runs Unit 3's Practice Problem 1 and asserts all thirteen frequent itemsets and all three strong rules.

Experiment 7 — Multilevel association rules

1. Question

Mine multilevel association rules over a product hierarchy.

2. Aim

Mine rules at more than one level of a hierarchy, and drop the redundant ones.

3. Steps

1. Try one support threshold at every level.
2. Lower the threshold at the deeper levels.
3. Drop a rule its ancestor already explains.
4. Check an ancestor's support against its children's.

4. Programme

"""Experiment 7 — Multilevel (generalized) association rules.

WEKA has NO built-in multilevel miner (see lab.md). The standard technique is
to ADD ANCESTOR ITEMS to each transaction and mine the extended data, then
discard redundant descendant rules afterwards.

Implements Unit 3 section 3.9: the uniform-support dilemma, reduced support,
and the redundant-ancestor test.
"""
import pandas as pd
from mlxtend.frequent_patterns import apriori, association_rules
from mlxtend.preprocessing import TransactionEncoder

# leaf -> (category, department)
TAXONOMY = {
    "amul_milk":       ("milk", "dairy"),
    "nandini_milk":    ("milk", "dairy"),
    "heritage_milk":   ("milk", "dairy"),
    "amul_cheese":     ("cheese", "dairy"),
    "britannia_bread": ("bread", "bakery"),
    "croissant":       ("pastry", "bakery"),
}

# Built so the redundancy test actually DISCRIMINATES:
#   amul_milk     6 baskets, 4 with bread  -> confidence 0.667
#   nandini_milk  6 baskets, 4 with bread  -> confidence 0.667
#   heritage_milk 4 baskets, 4 with bread  -> confidence 1.000
#   milk overall 16 baskets, 12 with bread -> confidence 0.750
# so the first two are redundant (deviation 0.083) and the third is not (0.25).
TRANSACTIONS = (
    [["amul_milk", "britannia_bread"]] * 4
    + [["amul_milk", "amul_cheese"]] * 2
    + [["nandini_milk", "britannia_bread"]] * 4
    + [["nandini_milk", "croissant"]] * 2
    + [["heritage_milk", "britannia_bread"]] * 4
    + [["amul_cheese", "croissant"]] * 2
    + [["amul_cheese", "britannia_bread"]] * 2
)


def expand(transaction, levels=(0, 1, 2)):
    """Add ancestor items. 0 = leaves, 1 = category, 2 = department."""
    out = set()
    for item in transaction:
        if 0 in levels:
            out.add(item)
        for lv in (1, 2):
            if lv in levels:
                out.add(TAXONOMY[item][lv - 1])
    return sorted(out)


def at_level(level):
    return [expand(t, levels=(level,)) for t in TRANSACTIONS]


def mine(transactions, min_support, min_confidence):
    te = TransactionEncoder()
    frame = pd.DataFrame(te.fit(transactions).transform(transactions),
                         columns=te.columns_)
    items = apriori(frame, min_support=min_support, use_colnames=True)
    if items.empty:
        return items, pd.DataFrame()
    rules = association_rules(items, metric="confidence", min_threshold=min_confidence)
    return items, rules


def confidence(antecedent, consequent, transactions):
    """Computed directly from the baskets, so nothing is hidden in a library."""
    a = sum(1 for t in transactions if antecedent in t)
    both = sum(1 for t in transactions if antecedent in t and consequent in t)
    return both / a if a else 0.0, a, both


def uniform_support_dilemma():
    """Section 3.9: ONE threshold cannot serve both levels."""
    leaf_items, _ = mine(at_level(0), 0.40, 0.5)
    leaf_pairs = [i for i in leaf_items.itemsets if len(i) >= 2]
    assert leaf_pairs == [], "at 40%, no individual PRODUCT pair qualifies"

    dept_items, _ = mine(at_level(2), 0.40, 0.5)
    dept_pairs = [i for i in dept_items.itemsets if len(i) >= 2]
    assert dept_pairs, "but dairy+bakery clears 40% comfortably"

    print(f"  uniform support 40%: {len(leaf_pairs)} product pairs, "
          f"{len(dept_pairs)} department pairs")
    print(f"       set it high and the leaves vanish; set it low and the top")
    print(f"       floods you -- which is why REDUCED support exists")


def reduced_support():
    """A lower threshold at deeper levels."""
    for name, level, minsup in [("department", 2, 0.40),
                                ("category", 1, 0.25),
                                ("product", 0, 0.15)]:
        items, rules = mine(at_level(level), minsup, 0.5)
        pairs = [tuple(sorted(i)) for i in items.itemsets if len(i) >= 2]
        print(f"  {name:11s} minsup {minsup:.0%}: {len(items):2d} itemsets, "
              f"{len(pairs)} of size >= 2, {len(rules)} rules")
        assert len(items) > 0, f"{name} level found nothing"


def redundant_ancestor_rule():
    """Section 3.9: report a descendant rule only when it DEVIATES.

    A descendant rule whose confidence matches the ancestor's tells you nothing
    new -- it is the EXPECTED consequence of the ancestor rule.
    """
    mixed = [expand(t) for t in TRANSACTIONS]        # leaves AND ancestors

    anc_conf, anc_n, anc_both = confidence("milk", "bread", mixed)
    assert (anc_n, anc_both) == (16, 12), (anc_n, anc_both)
    assert round(anc_conf, 4) == 0.75, anc_conf

    print(f"\n  ancestor: milk -> bread, {anc_both}/{anc_n} = {anc_conf:.4f}")
    print(f"  descendants (tolerance 0.10):")

    TOLERANCE = 0.10
    verdicts = {}
    for brand in ["amul_milk", "nandini_milk", "heritage_milk"]:
        conf, n, both = confidence(brand, "bread", mixed)
        deviation = abs(conf - anc_conf)
        redundant = deviation <= TOLERANCE
        verdicts[brand] = redundant
        print(f"    {brand:14s} -> bread: {both}/{n} = {conf:.4f}, "
              f"deviation {deviation:.4f} -> "
              f"{'REDUNDANT, suppress' if redundant else 'DEVIATES, report it'}")

    # The demonstration must DISCRIMINATE, or it proves nothing.
    assert verdicts["amul_milk"] is True
    assert verdicts["nandini_milk"] is True
    assert verdicts["heritage_milk"] is False
    assert any(verdicts.values()) and not all(verdicts.values()), \
        "the test must both suppress and report, or it demonstrates nothing"

    print(f"       two suppressed, one reported -- heritage_milk buyers really")
    print(f"       do buy bread more often than milk buyers in general")


def ancestor_support_is_at_least_the_sum():
    """Section 3.9: an ancestor's support is AT LEAST the sum of its
    descendants' -- at least, not equal, because one basket containing two
    brands of milk counts twice at the leaf level and once at 'milk'."""
    mixed = [expand(t) for t in TRANSACTIONS]
    n = len(mixed)

    milk = sum(1 for t in mixed if "milk" in t) / n
    brands = sum(sum(1 for t in mixed if b in t) for b in
                 ["amul_milk", "nandini_milk", "heritage_milk"]) / n

    assert milk <= brands + 1e-12, "the ancestor cannot exceed the leaf sum"
    assert milk >= max(sum(1 for t in mixed if b in t) / n for b in
                       ["amul_milk", "nandini_milk", "heritage_milk"])

    print(f"\n  monotonicity: support(milk) = {milk:.4f}, "
          f"sum of brand supports = {brands:.4f}")
    print(f"       so an infrequent ancestor prunes its whole subtree")


def main():
    print("Experiment 7 -- Multilevel association rules")
    # Step 1: Try one support threshold at every level
    uniform_support_dilemma()
    # Step 2: Lower the threshold at the deeper levels
    reduced_support()
    # Step 3: Drop a rule its ancestor already explains
    redundant_ancestor_rule()
    # Step 4: Check an ancestor's support against its children's
    ancestor_support_is_at_least_the_sum()
    print("\n  multilevel mining and the redundancy test verified")


if __name__ == "__main__":
    main()

5. Execution and Results

Experiment 7 -- Multilevel association rules
/tmp/claude-0/-home-user-planning-for-future/f2d15764-a81e-5002-9407-be4e7330bada/scratchpad/labenv/lib/python3.11/site-packages/mlxtend/frequent_patterns/association_rules.py:184: RuntimeWarning: invalid value encountered in divide
  cert_metric = np.where(certainty_denom == 0, 0, certainty_num / certainty_denom)
  uniform support 40%: 0 product pairs, 1 department pairs
       set it high and the leaves vanish; set it low and the top
       floods you -- which is why REDUCED support exists
/tmp/claude-0/-home-user-planning-for-future/f2d15764-a81e-5002-9407-be4e7330bada/scratchpad/labenv/lib/python3.11/site-packages/mlxtend/frequent_patterns/association_rules.py:184: RuntimeWarning: invalid value encountered in divide
  cert_metric = np.where(certainty_denom == 0, 0, certainty_num / certainty_denom)
  department  minsup 40%:  3 itemsets, 1 of size >= 2, 2 rules
  category    minsup 25%:  4 itemsets, 1 of size >= 2, 2 rules
  product     minsup 15%:  9 itemsets, 3 of size >= 2, 3 rules

  ancestor: milk -> bread, 12/16 = 0.7500
  descendants (tolerance 0.10):
    amul_milk      -> bread: 4/6 = 0.6667, deviation 0.0833 -> REDUNDANT, suppress
    nandini_milk   -> bread: 4/6 = 0.6667, deviation 0.0833 -> REDUNDANT, suppress
    heritage_milk  -> bread: 4/4 = 1.0000, deviation 0.2500 -> DEVIATES, report it
       two suppressed, one reported -- heritage_milk buyers really
       do buy bread more often than milk buyers in general

  monotonicity: support(milk) = 0.8000, sum of brand supports = 0.8000
       so an infrequent ancestor prunes its whole subtree

  multilevel mining and the redundancy test verified

There is no _weka.sh for this experiment: there is no WEKA class to run.

07_multilevel.py builds a small product taxonomy, expands each transaction with its ancestors, mines at two levels with different thresholds, and demonstrates the redundancy test: a descendant rule is reported only when its confidence deviates from what the ancestor rule predicts.

Experiment 8 — K-Means clustering

1. Question

Cluster a dataset with K-Means, and compare the clusters with the classes.

2. Aim

Run K-Means in WEKA, read its centroids and error, and see the seed change the answer.

3. Steps

In WEKA, from the command line, 08_kmeans_weka.sh:

1. Cluster iris into 3, ignoring the class, then compare with it (SimpleKMeans, classes to clusters).
2. The same with another seed, which can change the answer (seed = 31). Most.

In Python, 08_kmeans.py:

1. Run K-Means by hand on the eight 1-D points.
2. Run it on the eight 2-D points of Practice Problem 1.
3. Check that scikit-learn agrees.
4. See the starting centroids change the answer.
5. Choose k by the elbow and the silhouette.

4. Programme

In WEKA, from the command line, 08_kmeans_weka.sh:

#!/usr/bin/env bash
# Experiment 8 in WEKA 3.8.7, from the command line: K-Means.
# Each command runs the WEKA class that the Explorer runs for the same choice, and prints
# what the Explorer shows in its output pane. The click-path is on the lab page.
# Needs tools/data-science/setup_weka.sh; run as  bash 08_kmeans_weka.sh
set -euo pipefail
source "$(dirname "$0")/weka.sh"

# Step 1: Cluster iris into 3, ignoring the class, then compare with it (SimpleKMeans, classes to clusters)
weka weka.clusterers.SimpleKMeans -N 3 -S 10 -t $DATA/iris.arff -c last

# Step 2: The same with another seed, which can change the answer (seed = 31). Most
# seeds find the clustering above; this one stops in a worse local optimum.
weka weka.clusterers.SimpleKMeans -N 3 -S 31 -t $DATA/iris.arff -c last

In Python, 08_kmeans.py:

"""Experiment 8 — K-Means clustering.

Reproduces Unit 5 section 5.2's 1-D trace and Practice Problem 1's 2-D trace
exactly, then demonstrates the elbow and silhouette methods on iris.
"""
import numpy as np
from sklearn.cluster import KMeans
from sklearn.datasets import load_iris
from sklearn.metrics import silhouette_score, davies_bouldin_score
from sklearn.preprocessing import StandardScaler


def kmeans_by_hand(points, centroids, max_iter=100):
    """Plain Lloyd's algorithm, so every iteration can be printed and checked.

    Ties go to the LOWER-indexed centroid, which is what the notes assume.
    """
    points = np.asarray(points, dtype=float)
    centroids = np.asarray(centroids, dtype=float)
    history = []

    for _ in range(max_iter):
        d = np.linalg.norm(points[:, None] - centroids[None], axis=-1)
        labels = d.argmin(axis=1)                       # argmin breaks ties low
        new = np.array([points[labels == k].mean(axis=0) for k in range(len(centroids))])
        history.append((labels.copy(), centroids.copy()))
        if np.allclose(new, centroids):
            return labels, centroids, history
        centroids = new
    return labels, centroids, history


def wcss(points, labels, centroids):
    points = np.asarray(points, dtype=float)
    return float(sum(((points[labels == k] - centroids[k]) ** 2).sum()
                     for k in range(len(centroids))))


def unit5_one_dimensional():
    """Section 5.2: points 2,4,10,12,3,20,30,11 with c = 2, 4."""
    X = np.array([[2], [4], [10], [12], [3], [20], [30], [11]], dtype=float)
    labels, cent, hist = kmeans_by_hand(X, [[2.0], [4.0]])

    c1 = sorted(X[labels == 0].ravel().tolist())
    c2 = sorted(X[labels == 1].ravel().tolist())
    assert c1 == [2, 3, 4], c1
    assert c2 == [10, 11, 12, 20, 30], c2
    assert round(float(cent[0][0]), 4) == 3.0
    assert round(float(cent[1][0]), 4) == 16.6

    w = wcss(X, labels, cent)
    assert round(w, 2) == 289.20, w

    # The outlier 30 alone contributes 62.5% of cluster 2's error.
    c2_err = float(((X[labels == 1] - cent[1]) ** 2).sum())
    share = (30 - 16.6) ** 2 / c2_err
    assert round(c2_err, 2) == 287.20
    assert round(share * 100, 1) == 62.5

    # Convergence took three passes: two that moved, one that confirmed.
    assert len(hist) == 3, f"the notes show three iterations, got {len(hist)}"

    print(f"  5.2 1-D: C1={c1} c=3.0, C2={c2} c=16.6, WCSS={w:.1f}")
    print(f"       the single outlier 30 contributes {share:.1%} of C2's error")


def unit5_practice_1():
    """Practice Problem 1: eight 2-D points, initial centroids A and C."""
    names = list("ABCDEFGH")
    X = np.array([[2, 10], [2, 5], [8, 4], [5, 8], [7, 5], [6, 4], [1, 2], [4, 9]],
                 dtype=float)
    labels, cent, hist = kmeans_by_hand(X, [[2.0, 10.0], [8.0, 4.0]])

    c1 = sorted(names[i] for i in range(8) if labels[i] == 0)
    c2 = sorted(names[i] for i in range(8) if labels[i] == 1)
    assert c1 == ["A", "B", "D", "H"], c1
    assert c2 == ["C", "E", "F", "G"], c2

    assert [round(v, 4) for v in cent[0]] == [3.25, 8.0], cent[0]
    assert [round(v, 4) for v in cent[1]] == [5.5, 3.75], cent[1]

    w = wcss(X, labels, cent)
    assert round(w, 2) == 54.50, w

    # G(1,2) is the far point: 42.8% of the total error.
    g_err = float(((X[6] - cent[1]) ** 2).sum())
    assert round(g_err, 4) == 23.3125
    assert round(g_err / w * 100, 1) == 42.8

    assert len(hist) == 2, "the notes show convergence after two iterations"

    print(f"  Practice 1: C1={c1} at (3.25, 8.0), C2={c2} at (5.5, 3.75)")
    print(f"       WCSS={w:.2f}; G alone contributes {g_err / w:.1%}")


def sklearn_agrees():
    """scikit-learn, given the same initial centroids, must land in the same place."""
    X = np.array([[2, 10], [2, 5], [8, 4], [5, 8], [7, 5], [6, 4], [1, 2], [4, 9]],
                 dtype=float)
    km = KMeans(n_clusters=2, init=np.array([[2.0, 10.0], [8.0, 4.0]]),
                n_init=1, random_state=0).fit(X)
    assert round(float(km.inertia_), 2) == 54.50, km.inertia_
    centres = sorted(tuple(round(v, 4) for v in c) for c in km.cluster_centers_)
    assert centres == [(3.25, 8.0), (5.5, 3.75)], centres
    print(f"  sklearn KMeans: inertia {km.inertia_:.2f} -- identical")


def initialisation_matters():
    """Section 5.2 weakness 2: K-Means finds only a LOCAL optimum."""
    rng = np.random.default_rng(0)
    X = np.vstack([rng.normal([0, 0], 0.5, (40, 2)),
                   rng.normal([5, 5], 0.5, (40, 2)),
                   rng.normal([0, 5], 0.5, (40, 2))])

    # Rounded to the 2 places printed: K-Means sums in parallel threads, so two
    # runs can differ in the last bits, and an unrounded set counted the same
    # optimum twice -- 6 distinct on one run, 7 on the next. [Corrected.]
    inertias = {round(KMeans(n_clusters=3, init="random", n_init=1,
                             random_state=s).fit(X).inertia_, 2) for s in range(30)}
    assert len(inertias) > 1, "different seeds must give different local optima"

    best_single = min(inertias)
    multi = KMeans(n_clusters=3, n_init=10, random_state=0).fit(X).inertia_
    assert multi <= max(inertias) + 0.005

    print(f"  initialisation: {len(inertias)} distinct optima across 30 seeds "
          f"({min(inertias):.2f} to {max(inertias):.2f})")
    print(f"       n_init=10 finds {multi:.2f} -- which is why it is the default")


def choosing_k():
    """Elbow and silhouette on iris."""
    X, y = load_iris(return_X_y=True)
    Xs = StandardScaler().fit_transform(X)

    print("      k    WCSS  silhouette  Davies-Bouldin")
    sil = {}
    prev = None
    for k in range(2, 8):
        km = KMeans(n_clusters=k, n_init=10, random_state=0).fit(Xs)
        s = silhouette_score(Xs, km.labels_)
        db = davies_bouldin_score(Xs, km.labels_)
        sil[k] = s
        print(f"      {k}  {km.inertia_:7.2f}     {s:.4f}        {db:.4f}")
        if prev is not None:
            assert km.inertia_ < prev, "WCSS must fall monotonically with k"
        prev = km.inertia_

    best = max(sil, key=sil.get)
    assert best == 2, f"silhouette peaks at k=2 on scaled iris, got {best}"

    print(f"       silhouette peaks at k={best}, though iris has 3 species --")
    print(f"       two of them overlap, so the DATA says 2 and the LABELS say 3.")
    print(f"       That is exactly why clustering has no single right answer.")


def main():
    print("Experiment 8 -- K-Means")
    # Step 1: Run K-Means by hand on the eight 1-D points
    unit5_one_dimensional()
    # Step 2: Run it on the eight 2-D points of Practice Problem 1
    unit5_practice_1()
    # Step 3: Check that scikit-learn agrees
    sklearn_agrees()
    # Step 4: See the starting centroids change the answer
    initialisation_matters()
    # Step 5: Choose k by the elbow and the silhouette
    choosing_k()
    print("  all Unit 5 K-Means calculations reproduced")


if __name__ == "__main__":
    main()