Practical Lab

18 practicals, each set out as 1. Question, 2. Aim, 3. Steps, 4. Programme, 5. Execution and Results.

Code lives in labs/course-9-python-da/.

pip install -r tools/data-science/requirements.txt
python3 tools/data-science/run_data_labs.py course9

Three practicals time something — 2's vectorisation and 12's apply against arithmetic, best of three runs where it says so. A timing measures the machine at a moment, so those lines differ from run to run, and capture_lab_outputs.py --check sets them aside; every other line must repeat exactly.

Working environment

The lab exam will give you either Jupyter or a plain editor.

jupyter lab          # or: jupyter notebook
python3 script.py

In Jupyter, three things save you time: df.<TAB> completes method names, pd.merge? shows the docstring, and %timeit expr measures a line. Nobody memorises the parameter lists — knowing how to look them up is the actual skill, and examiners know it.

One import block for everything:

import numpy as np
import pandas as pd
import matplotlib
matplotlib.use("Agg")          # no display on a server; saves files instead
import matplotlib.pyplot as plt
import seaborn as sns

pd.set_option("display.max_columns", None)
pd.set_option("display.width", 120)

Practical 1 — Create and manipulate ndarrays; explore data types

1. Question

Create NumPy arrays and explore their attributes and data types.

2. Aim

Make arrays in the standard ways, read their attributes, and meet the dtype traps.

3. Steps

1. Create arrays.
2. Read their attributes.
3. Meet the dtype traps.
4. See that np.empty is not zeros.

a = np.array([1, 2, 3])
b = np.zeros((2, 3));  c = np.ones((2, 3));  d = np.full((2, 3), 7)
e = np.arange(0, 10, 2);  f = np.linspace(0, 1, 5)
g = np.eye(3);  h = np.random.default_rng(42).random((2, 3))

a.ndim, a.shape, a.size, a.dtype, a.itemsize, a.nbytes
a.astype(np.float64)

4. Programme

"""Practical 1 — Create and manipulate NumPy ndarrays; explore data types."""
import numpy as np


def creating():
    a = np.array([1, 2, 3])
    assert a.tolist() == [1, 2, 3]

    assert np.zeros(5).tolist() == [0.0] * 5
    assert np.zeros((2, 3)).shape == (2, 3)
    assert np.ones((2, 3)).sum() == 6
    assert np.full((2, 3), 7).ravel().tolist() == [7] * 6
    assert np.eye(3).trace() == 3.0
    assert np.diag([1, 2, 3]).sum() == 6

    assert np.arange(10).tolist() == list(range(10))
    assert np.arange(2, 10, 2).tolist() == [2, 4, 6, 8]
    assert np.linspace(0, 1, 5).tolist() == [0.0, 0.25, 0.5, 0.75, 1.0]

    # arange with a float step: the COUNT is not always what the arithmetic
    # suggests, because of floating-point accumulation. linspace asks for a
    # count and delivers it.
    assert np.linspace(0, 1, 11).size == 11
    assert np.arange(0, 0.3, 0.1).size == 3

    rng = np.random.default_rng(42)
    r = rng.random((2, 3))
    assert r.shape == (2, 3) and (0 <= r).all() and (r < 1).all()

    print("  creation: zeros, ones, full, eye, arange, linspace, rng -- all as documented")


def attributes():
    a = np.array([[1, 2], [3, 4]])
    assert a.ndim == 2
    assert a.shape == (2, 2)
    assert a.size == 4
    assert a.dtype == np.int64
    assert a.itemsize == 8
    assert a.nbytes == 32

    # A 1-D array's shape is a 1-TUPLE, not an int and not (n, 1).
    assert np.array([1, 2, 3]).shape == (3,)
    assert np.array([1, 2, 3]).reshape(-1, 1).shape == (3, 1)

    print(f"  attributes: ndim {a.ndim}, shape {a.shape}, size {a.size}, "
          f"itemsize {a.itemsize}, nbytes {a.nbytes}")


def dtype_traps():
    # 1. Integer overflow WRAPS, silently.
    assert (np.array([127], dtype=np.int8) + 1)[0] == -128

    # 2. Assigning a float into an int array TRUNCATES, silently.
    a = np.array([1, 2, 3])
    a[0] = 3.7
    assert a.tolist() == [3, 2, 3]

    # ...and then true division still gives float64
    assert (a / 2).tolist() == [1.5, 1.0, 1.5]
    assert (a // 2).tolist() == [1, 1, 1]

    # 3. One string makes EVERYTHING a string.
    assert np.array([1, 2, "3"]).dtype.str == "<U21"

    # 4. Mixing int and float upcasts to float.
    assert np.array([1, 2, 3.5]).dtype == np.float64

    # astype always COPIES
    b = np.array([1, 2, 3])
    c = b.astype(np.float64)
    c[0] = 99
    assert b[0] == 1, "astype must not alias"

    print("  dtypes: int8 127+1 -> -128; 3.7 into int -> 3; one string -> all strings")


def empty_is_not_zeros():
    """np.empty hands you whatever was in that memory."""
    e = np.empty((2, 3))
    assert e.shape == (2, 3)
    # We cannot assert its CONTENTS -- that is exactly the point.
    z = np.zeros((2, 3))
    assert z.sum() == 0.0
    print("  np.empty allocates without zeroing -- faster, and a bug if unfilled")


def main():
    print("Practical 1 -- ndarray basics and dtypes")
    # Step 1: Create arrays
    creating()
    # Step 2: Read their attributes
    attributes()
    # Step 3: Meet the dtype traps
    dtype_traps()
    # Step 4: See that np.empty is not zeros
    empty_is_not_zeros()


if __name__ == "__main__":
    main()

5. Execution and Results

Practical 1 -- ndarray basics and dtypes
  creation: zeros, ones, full, eye, arange, linspace, rng -- all as documented
  attributes: ndim 2, shape (2, 2), size 4, itemsize 8, nbytes 32
  dtypes: int8 127+1 -> -128; 3.7 into int -> 3; one string -> all strings
  np.empty allocates without zeroing -- faster, and a bug if unfilled

Practical 2 — Arithmetic and element-wise calculations

1. Question

Perform arithmetic and element-wise calculations on arrays.

2. Aim

Compute on whole arrays at once, broadcast shapes together, and measure what that saves.

3. Steps

1. Compare a list with an array.
2. Compute element-wise.
3. Broadcast shapes together.
4. Reduce along an axis.
5. Time a loop against a vectorised sum.

a - a.mean(axis=0)                    # centre each column  ✓
a - a.mean(axis=1)                    # ValueError -- shapes (2,3) and (2,)
a - a.mean(axis=1, keepdims=True)     # ✓

4. Programme

"""Practical 2 — Arithmetic, broadcasting and why vectorising matters."""
import time
import numpy as np


def list_versus_array():
    """The first thing to get right coming from Course 3."""
    assert [1, 2, 3] + [4, 5, 6] == [1, 2, 3, 4, 5, 6], "lists CONCATENATE"
    assert (np.array([1, 2, 3]) + np.array([4, 5, 6])).tolist() == [5, 7, 9], "arrays ADD"

    assert [1, 2, 3] * 2 == [1, 2, 3, 1, 2, 3], "lists REPEAT"
    assert (np.array([1, 2, 3]) * 2).tolist() == [2, 4, 6], "arrays SCALE"

    print("  + concatenates lists and ADDS arrays; * repeats lists and SCALES arrays")


def elementwise():
    a = np.array([1, 2, 3, 4])
    b = np.array([10, 20, 30, 40])
    assert (a + b).tolist() == [11, 22, 33, 44]
    assert (a * b).tolist() == [10, 40, 90, 160]
    assert (b / a).tolist() == [10.0] * 4
    assert (a ** 2).tolist() == [1, 4, 9, 16]
    assert (a > 2).tolist() == [False, False, True, True]
    print("  element-wise arithmetic and comparison, no loop anywhere")


def broadcasting():
    a = np.array([[1, 2, 3], [4, 5, 6]])          # (2, 3)

    assert (a + 10).tolist() == [[11, 12, 13], [14, 15, 16]]
    assert (a + np.array([10, 20, 30])).tolist() == [[11, 22, 33], [14, 25, 36]]
    assert (a + np.array([[10], [20]])).tolist() == [[11, 12, 13], [24, 25, 26]]

    # Centring each COLUMN works: mean(axis=0) has shape (3,)
    assert a.mean(axis=0).shape == (3,)
    assert (a - a.mean(axis=0)).shape == (2, 3)

    # Centring each ROW does NOT, without keepdims -- and seeing the error is
    # what makes keepdims memorable.
    assert a.mean(axis=1).shape == (2,)
    try:
        a - a.mean(axis=1)
        raise AssertionError("expected a broadcasting ValueError")
    except ValueError as e:
        assert "broadcast" in str(e).lower() or "shape" in str(e).lower()

    assert a.mean(axis=1, keepdims=True).shape == (2, 1)
    assert (a - a.mean(axis=1, keepdims=True)).shape == (2, 3)
    assert (a - a.mean(axis=1)[:, np.newaxis]).shape == (2, 3)

    print("  broadcasting: (2,3)+(3,) per row, (2,3)+(2,1) per column")
    print("       centring rows NEEDS keepdims=True -- ValueError without it")


def axis_is_the_one_that_disappears():
    a = np.array([[1, 2, 3], [4, 5, 6]])
    assert a.sum() == 21
    assert a.sum(axis=0).tolist() == [5, 7, 9], "axis=0 collapses ROWS -> per COLUMN"
    assert a.sum(axis=1).tolist() == [6, 15], "axis=1 collapses COLUMNS -> per ROW"
    assert a.sum(axis=0).shape == (3,)
    assert a.sum(axis=1).shape == (2,)
    print("  axis=0 -> [5 7 9] (one per column); axis=1 -> [6 15] (one per row)")


def vectorising_is_faster():
    """Best of three, to avoid an unwarmed single run misleading us."""
    n = 1_000_000
    lst = list(range(n))
    arr = np.arange(n, dtype=np.float64)
    lstf = [float(v) for v in lst]

    def best(fn, reps=3):
        t = float("inf")
        for _ in range(reps):
            s = time.perf_counter()
            fn()
            t = min(t, time.perf_counter() - s)
        return t

    cases = [
        ("x * 2",       lambda: [v * 2 for v in lstf],  lambda: arr * 2),
        ("sqrt(x)",     lambda: [v ** 0.5 for v in lstf], lambda: np.sqrt(arr)),
        ("dot product", lambda: sum(x * y for x, y in zip(lstf, lstf)),
                        lambda: arr @ arr),
    ]

    print(f"  vectorisation on {n:,} elements (best of 3)")
    for name, py, npf in cases:
        # Correctness first. This is the assertion that means something: the
        # two paths must agree, or the comparison is between two different
        # calculations and the timing is meaningless.
        py_result, np_result = py(), npf()
        assert np.allclose(np.asarray(py_result), np.asarray(np_result)), name

        p, q = best(py), best(npf)
        speedup = p / q
        print(f"    {name:12s} python {p*1000:7.1f} ms   numpy {q*1000:6.2f} ms"
              f"   {speedup:6.1f}x")

        # Only that NumPy wins is asserted, not by how much.
        #
        # This assertion used to demand a 10x floor, and it failed on a loaded
        # machine at 7.1x for the dot product -- a false alarm, because a wall
        # clock measures the machine, not the code. A threshold that fails
        # when nothing is wrong teaches the reader to ignore failures, which
        # is worse than having no threshold at all. So the factor is reported,
        # not asserted, and the number you see is the one this run measured.
        assert speedup > 1, f"{name} was not faster in NumPy ({speedup:.1f}x)"

    # A note on the dot product, because its number moves the most. `arr @ arr`
    # goes to BLAS, which is threaded and memory-bandwidth-bound, so what it
    # measures is partly the machine: this same line has been seen at 7x with
    # other work running on the box and at over 300x on an idle one, on the
    # same data. The two element-wise cases are far steadier because they are
    # dominated by allocating a million-element Python list.
    #
    # That spread is the argument against asserting a threshold here. It is
    # also the more useful lesson than any single figure: a timing is a
    # measurement of a machine at a moment, and quoting one as though it were
    # a property of the code is how benchmarks mislead.


def main():
    print("Practical 2 -- Arithmetic and broadcasting")
    # Step 1: Compare a list with an array
    list_versus_array()
    # Step 2: Compute element-wise
    elementwise()
    # Step 3: Broadcast shapes together
    broadcasting()
    # Step 4: Reduce along an axis
    axis_is_the_one_that_disappears()
    # Step 5: Time a loop against a vectorised sum
    vectorising_is_faster()


if __name__ == "__main__":
    main()

5. Execution and Results

Practical 2 -- Arithmetic and broadcasting
  + concatenates lists and ADDS arrays; * repeats lists and SCALES arrays
  element-wise arithmetic and comparison, no loop anywhere
  broadcasting: (2,3)+(3,) per row, (2,3)+(2,1) per column
       centring rows NEEDS keepdims=True -- ValueError without it
  axis=0 -> [5 7 9] (one per column); axis=1 -> [6 15] (one per row)
  vectorisation on 1,000,000 elements (best of 3)
    x * 2        python    47.2 ms   numpy   0.80 ms     59.4x
    sqrt(x)      python    58.7 ms   numpy   0.98 ms     59.8x
    dot product  python    29.6 ms   numpy   0.13 ms    225.8x

The speed measurement. The script times a Python comprehension against the vectorised form, best of three runs, on 1,000,000 elements; the figures above are this run's. It asserts only that NumPy is faster — the program's own comments say why: a fixed floor (it was 10×) failed on a loaded machine at 7.1× for the dot product, a false alarm, because a wall clock measures the machine, not the code.

Corrected: this page said the script asserts a speed-up of more than 10×, and gave a table of approximate timings (~52×, ~43×, ~440×). The assertion was relaxed to "faster" when the floor proved flaky, and timings vary from run to run; the run's own are above.

Practical 3 — Indexing, slicing, boolean and fancy indexing

1. Question

Select elements of arrays by indexing, slicing, boolean masks and lists of indices.

2. Aim

Select from arrays every way NumPy allows, and know which ways copy.

3. Steps

1. Slice.
2. Tell a view from a copy.
3. Select with a boolean mask.
4. See why and raises.
5. Select with lists of indices.
6. Reshape.

a[1:4]                   # VIEW  -- a[1:4].base is a
a[a > 2]                 # COPY
a[[0, 2]]                # COPY

4. Programme

"""Practical 3 — Indexing, slicing, boolean and fancy indexing."""
import numpy as np


def basic_slicing():
    a = np.arange(10)
    assert a[0] == 0 and a[-1] == 9
    assert a[2:5].tolist() == [2, 3, 4]
    assert a[:3].tolist() == [0, 1, 2]
    assert a[::2].tolist() == [0, 2, 4, 6, 8]
    assert a[::-1].tolist() == list(range(9, -1, -1))

    m = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
    assert m[1, 2] == 6
    assert m[1][2] == 6, "works, but builds a temporary row first"
    assert m[1].tolist() == [4, 5, 6]
    assert m[:, 1].tolist() == [2, 5, 8]
    assert m[0:2, 1:3].tolist() == [[2, 3], [5, 6]]
    assert m[::2, ::2].tolist() == [[1, 3], [7, 9]]
    print("  slicing: comma for 2-D, m[:, 1] for a column, strides work per axis")


def views_and_copies():
    """The behaviour that differs from Python lists and causes real bugs."""
    lst = [1, 2, 3, 4, 5]
    s = lst[1:4]
    s[0] = 99
    assert lst == [1, 2, 3, 4, 5], "list slicing COPIES"

    a = np.array([1, 2, 3, 4, 5])
    v = a[1:4]
    v[0] = 99
    assert a.tolist() == [1, 99, 3, 4, 5], "array slicing is a VIEW"
    assert a[1:4].base is a, "a view knows its parent"

    b = np.array([1, 2, 3, 4, 5])
    f = b[[0, 2]]
    f[0] = 99
    assert b.tolist() == [1, 2, 3, 4, 5], "fancy indexing COPIES"
    assert b[[0, 2]].base is not b

    c = np.array([1, 2, 3, 4, 5])
    mask = c[c > 2]
    mask[0] = 99
    assert c.tolist() == [1, 2, 3, 4, 5], "boolean masking COPIES"

    d = np.array([1, 2, 3, 4, 5])
    e = d[1:4].copy()
    e[0] = 99
    assert d.tolist() == [1, 2, 3, 4, 5], ".copy() opts out of the view"

    print("  slice -> VIEW; boolean and fancy -> COPY; .copy() when you mean it")


def boolean_indexing():
    marks = np.array([72, 45, 91, 66, 38, 88])

    assert (marks > 50).tolist() == [True, False, True, True, False, True]
    assert marks[marks > 50].tolist() == [72, 91, 66, 88]
    assert marks[marks > 50].mean() == 79.25
    assert marks[(marks > 50) & (marks < 90)].tolist() == [72, 66, 88]
    assert sorted(marks[(marks < 40) | (marks > 90)].tolist()) == [38, 91]
    assert marks[~(marks > 50)].tolist() == [45, 38]

    assert (marks > 50).sum() == 4, "True counts as 1"
    assert round(float((marks > 50).mean()), 4) == 0.6667, "the PROPORTION"
    assert (marks > 50).any() and not (marks > 50).all()
    assert np.where(marks > 50)[0].tolist() == [0, 2, 3, 5], "the INDICES"

    graded = np.where(marks >= 50, "Pass", "Fail")
    assert graded.tolist() == ["Pass", "Fail", "Pass", "Pass", "Fail", "Pass"]

    print("  boolean: (cond).sum() counts, (cond).mean() gives the proportion")


def and_raises():
    """Python's `and` cannot reduce an array to one truth value."""
    marks = np.array([72, 45, 91, 66, 38, 88])
    try:
        marks[(marks > 50) and (marks < 90)]
        raise AssertionError("expected ValueError from `and`")
    except ValueError as e:
        assert "ambiguous" in str(e)
    print("  `and` raises 'truth value is ambiguous' -- use & with parentheses")


def fancy_indexing():
    a = np.array([10, 20, 30, 40, 50])
    assert a[[0, 2, 4]].tolist() == [10, 30, 50]
    assert a[[4, 4, 0]].tolist() == [50, 50, 10], "repeats and any order allowed"

    m = np.arange(12).reshape(3, 4)
    assert m[[0, 2]].tolist() == [[0, 1, 2, 3], [8, 9, 10, 11]]

    # Two index arrays are PAIRED position by position -- three elements,
    # not a 3x3 block. np.ix_ is what gives the submatrix.
    assert m[[0, 1, 2], [1, 2, 3]].tolist() == [1, 6, 11]
    assert m[np.ix_([0, 2], [1, 3])].tolist() == [[1, 3], [9, 11]]
    assert m[np.ix_([0, 2], [1, 3])].shape == (2, 2)

    print("  fancy: m[[0,1,2],[1,2,3]] pairs -> 3 elements; np.ix_ -> the 2x2 block")


def reshaping():
    a = np.arange(12)
    assert a.reshape(3, 4).shape == (3, 4)
    assert a.reshape(3, -1).shape == (3, 4), "-1 means 'work it out'"
    assert a.reshape(-1, 1).shape == (12, 1)

    try:
        a.reshape(5, 3)
        raise AssertionError("expected a size mismatch")
    except ValueError:
        pass

    m = np.arange(6).reshape(2, 3)
    assert m.T.shape == (3, 2)
    assert m.T.base is not None, ".T is a VIEW -- transposing costs nothing"
    assert np.swapaxes(m, 0, 1).tolist() == m.T.tolist()

    t = np.arange(24).reshape(2, 3, 4)
    assert t.transpose(1, 0, 2).shape == (3, 2, 4)
    assert np.swapaxes(t, 0, 2).shape == (4, 3, 2)

    # concatenate joins along an EXISTING axis; stack ADDS one.
    x, y = np.array([1, 2]), np.array([3, 4])
    assert np.concatenate([x, y]).shape == (4,)
    assert np.stack([x, y]).shape == (2, 2)

    print("  reshape(-1) infers; .T is a view; concatenate keeps ndim, stack adds one")


def main():
    print("Practical 3 -- Indexing and slicing")
    # Step 1: Slice
    basic_slicing()
    # Step 2: Tell a view from a copy
    views_and_copies()
    # Step 3: Select with a boolean mask
    boolean_indexing()
    # Step 4: See why `and` raises
    and_raises()
    # Step 5: Select with lists of indices
    fancy_indexing()
    # Step 6: Reshape
    reshaping()


if __name__ == "__main__":
    main()

5. Execution and Results

Practical 3 -- Indexing and slicing
  slicing: comma for 2-D, m[:, 1] for a column, strides work per axis
  slice -> VIEW; boolean and fancy -> COPY; .copy() when you mean it
  boolean: (cond).sum() counts, (cond).mean() gives the proportion
  `and` raises 'truth value is ambiguous' -- use & with parentheses
  fancy: m[[0,1,2],[1,2,3]] pairs -> 3 elements; np.ix_ -> the 2x2 block
  reshape(-1) infers; .T is a view; concatenate keeps ndim, stack adds one

Practical 4 — Universal functions and statistics

1. Question

Apply universal functions and mathematical and statistical functions to arrays.

2. Aim

Use NumPy's ufuncs and statistics, and know where its defaults differ from Pandas.

3. Steps

1. Apply the unary ufuncs.
2. Tell np.maximum from np.max.
3. See NaN propagate.
4. Compute the statistics.
5. Set ddof, NumPy's against Pandas's.
6. Generate random numbers.

np.sqrt, np.exp, np.log, np.abs, np.round, np.sin
np.maximum(a, b)      # ELEMENT-WISE pairing
np.max(a)             # the largest ONE value
a.sum(axis=0)  a.mean(axis=1)  a.std(ddof=1)  a.argmax()  a.cumsum()

4. Programme

"""Practical 4 — Universal functions and mathematical/statistical functions."""
import numpy as np
import pandas as pd
from fixtures import COURSE4_SAMPLE


def unary_ufuncs():
    a = np.array([1.0, 4.0, 9.0, 16.0])
    assert np.sqrt(a).tolist() == [1.0, 2.0, 3.0, 4.0]
    assert np.abs(np.array([-1, -2, 3])).tolist() == [1, 2, 3]
    assert np.sign(np.array([-3, 0, 5])).tolist() == [-1, 0, 1]

    b = np.array([1.2, 1.5, 1.8, -1.5])
    assert np.floor(b).tolist() == [1.0, 1.0, 1.0, -2.0]
    assert np.ceil(b).tolist() == [2.0, 2.0, 2.0, -1.0]
    assert np.trunc(b).tolist() == [1.0, 1.0, 1.0, -1.0]

    assert round(float(np.exp(1)), 6) == 2.718282
    assert np.log(np.e) == 1.0, "np.log is the NATURAL logarithm"
    assert np.log10(1000.0) == 3.0
    assert np.log2(8.0) == 3.0
    assert round(float(np.sin(np.pi / 2)), 10) == 1.0, "radians, not degrees"

    print("  unary ufuncs: sqrt, floor/ceil/trunc, exp, log (NATURAL), sin (radians)")


def maximum_versus_max():
    """A binary ufunc against a reduction -- routinely confused."""
    assert np.maximum([1, 5, 3], [4, 2, 6]).tolist() == [4, 5, 6], "PAIRWISE"
    assert np.max([1, 5, 3]) == 5, "the largest ONE value"
    assert np.minimum([1, 5, 3], [4, 2, 6]).tolist() == [1, 2, 3]
    assert np.min([1, 5, 3]) == 1
    print("  np.maximum pairs element-wise; np.max reduces to one value")


def nan_propagates():
    a = np.array([1.0, np.nan, 3.0])
    assert np.isnan(a.sum()), "one NaN poisons the whole reduction"
    assert np.nansum(a) == 4.0
    assert np.nanmean(a) == 2.0
    assert np.isnan(a).tolist() == [False, True, False]
    print("  NaN propagates through sum(); nansum/nanmean skip it")


def statistics():
    a = np.array([[1, 2, 3], [4, 5, 6]])
    assert a.sum() == 21
    assert a.sum(axis=0).tolist() == [5, 7, 9]
    assert a.sum(axis=1).tolist() == [6, 15]
    assert a.mean() == 3.5
    assert a.min() == 1 and a.max() == 6
    assert a.argmin() == 0 and a.argmax() == 5, "argmin/argmax give the INDEX"
    assert a.cumsum().tolist() == [1, 3, 6, 10, 15, 21]
    assert np.median(a) == 3.5
    assert round(float(a.std()), 4) == 1.7078
    assert round(float(a.var()), 4) == 2.9167

    b = np.array([3, 1, 2])
    assert np.sort(b).tolist() == [1, 2, 3]
    assert b.tolist() == [3, 1, 2], "np.sort returns a COPY"
    assert np.argsort(b).tolist() == [1, 2, 0]
    b.sort()
    assert b.tolist() == [1, 2, 3], "the METHOD sorts in place"

    assert np.unique(np.array([3, 1, 3, 2, 1])).tolist() == [1, 2, 3]

    print(f"  statistics: sum {a.sum()}, per-column {a.sum(axis=0).tolist()}, "
          f"argmax {a.argmax()}, std {a.std():.4f}")


def ddof_is_the_examinable_one():
    """NumPy defaults to the POPULATION formula; Pandas to the SAMPLE one."""
    x = COURSE4_SAMPLE.to_numpy()
    assert x.tolist() == [2, 4, 4, 4, 5, 5, 7, 9]

    np_pop = float(np.std(x))
    np_sam = float(np.std(x, ddof=1))
    pd_default = float(pd.Series(x).std())

    assert np_pop == 2.0, "numpy default: ddof=0, POPULATION"
    assert round(np_sam, 4) == 2.1381, "ddof=1: SAMPLE"
    assert round(pd_default, 4) == 2.1381, "pandas default: ddof=1, SAMPLE"
    assert np_pop != pd_default, "the two libraries DISAGREE by default"

    # Course 4 taught the sample formula, so Pandas agrees and NumPy does not.
    assert round(pd_default, 4) == round(np_sam, 4)

    print(f"  ddof: np.std {np_pop} (population) vs pd.Series.std "
          f"{pd_default:.4f} (sample)")
    print(f"       SAME DATA, TWO LIBRARIES, TWO ANSWERS -- pass ddof explicitly")


def random_generation():
    rng = np.random.default_rng(42)
    r1 = rng.random(5)
    rng2 = np.random.default_rng(42)
    assert np.allclose(r1, rng2.random(5)), "same seed, same numbers"

    rng = np.random.default_rng(0)
    dice = rng.integers(1, 7, size=1000)
    assert dice.min() >= 1 and dice.max() <= 6, "high is EXCLUSIVE"

    # Two dice: P(sum = 7) is 6/36 exactly.
    rng = np.random.default_rng(0)
    rolls = rng.integers(1, 7, size=(10_000, 2))
    p7 = float((rolls.sum(axis=1) == 7).mean())
    exact = 6 / 36
    assert abs(p7 - exact) < 0.01, f"{p7} vs {exact}"

    print(f"  P(sum=7) simulated {p7:.4f} vs exact {exact:.4f} -- Course 4's")
    print(f"       classical probability confirmed empirically")


def main():
    print("Practical 4 -- Universal functions and statistics")
    # Step 1: Apply the unary ufuncs
    unary_ufuncs()
    # Step 2: Tell np.maximum from np.max
    maximum_versus_max()
    # Step 3: See NaN propagate
    nan_propagates()
    # Step 4: Compute the statistics
    statistics()
    # Step 5: Set ddof, NumPy's against Pandas's
    ddof_is_the_examinable_one()
    # Step 6: Generate random numbers
    random_generation()


if __name__ == "__main__":
    main()

5. Execution and Results

Practical 4 -- Universal functions and statistics
  unary ufuncs: sqrt, floor/ceil/trunc, exp, log (NATURAL), sin (radians)
  np.maximum pairs element-wise; np.max reduces to one value
  NaN propagates through sum(); nansum/nanmean skip it
  statistics: sum 21, per-column [5, 7, 9], argmax 5, std 1.7078
  ddof: np.std 2.0 (population) vs pd.Series.std 2.1381 (sample)
       SAME DATA, TWO LIBRARIES, TWO ANSWERS -- pass ddof explicitly
  P(sum=7) simulated 0.1697 vs exact 0.1667 -- Course 4's
       classical probability confirmed empirically

Practical 5 — Create and manipulate Series and DataFrames

1. Question

Create and manipulate Pandas Series and DataFrames.

2. Aim

Build Series and DataFrames from dicts and lists, and inspect and extend them.

3. Steps

1. Create a Series.
2. Create a DataFrame.
3. Inspect it.
4. Work with Index objects.
5. Add columns.

pd.Series([72,45,91], index=["Asha","Ravi","Meena"], name="marks")
pd.Series({"a": 1, "b": 2})            # from a dict
pd.DataFrame({"a":[1,2], "b":[3,4]})   # dict -> COLUMNS
pd.DataFrame([[1,3],[2,4]], columns=["a","b"])   # list -> ROWS

4. Programme

"""Practical 5 — Create and manipulate Pandas Series and DataFrames."""
import numpy as np
import pandas as pd
from fixtures import students


def series_creation():
    s = pd.Series([72, 45, 91, 66],
                  index=["Asha", "Ravi", "Meena", "Kiran"], name="marks")
    assert s.shape == (4,)
    assert s.name == "marks"
    assert s.index.tolist() == ["Asha", "Ravi", "Meena", "Kiran"]
    assert s["Asha"] == 72, "by LABEL"
    assert s.iloc[0] == 72, "by POSITION"
    assert "Asha" in s, "`in` checks the INDEX, like a dict"
    assert s[s > 60].tolist() == [72, 91, 66]

    assert pd.Series([1, 2, 3]).index.tolist() == [0, 1, 2], "default RangeIndex"
    assert pd.Series({"a": 1, "b": 2}).index.tolist() == ["a", "b"], "dict keys"
    assert pd.Series(5, index=["a", "b", "c"]).tolist() == [5, 5, 5], "scalar broadcasts"

    print(f"  Series: {s.shape[0]} labelled values, array-like and dict-like at once")


def dataframe_creation():
    """A dict gives COLUMNS; a list of lists gives ROWS."""
    by_cols = pd.DataFrame({"a": [1, 2], "b": [3, 4]})
    by_rows = pd.DataFrame([[1, 3], [2, 4]], columns=["a", "b"])
    by_dicts = pd.DataFrame([{"a": 1, "b": 3}, {"a": 2, "b": 4}])

    assert by_cols.equals(by_rows), "the same table, built two ways"
    assert by_cols.equals(by_dicts)
    assert by_cols.shape == (2, 2)

    # Getting it the wrong way round transposes the table.
    wrong = pd.DataFrame([[1, 2], [3, 4]], columns=["a", "b"])
    assert wrong.a.tolist() == [1, 3], "rows, not columns"
    assert not wrong.equals(by_cols) or True

    print("  DataFrame: dict -> columns, list of lists -> rows, list of dicts -> rows")


def attributes_and_inspection():
    df = students()
    assert df.shape == (5, 5), "5 students, 5 columns"
    assert df.size == 25
    assert list(df.columns) == ["roll", "name", "dept", "maths", "stats"]
    assert df.index.tolist() == [0, 1, 2, 3, 4]

    assert len(df.head(2)) == 2
    assert len(df.tail(3)) == 3
    assert df.dept.value_counts().to_dict() == {"DS": 3, "Stats": 2}
    assert df.nunique().dept == 2
    assert df.isna().sum().sum() == 0

    d = df.describe()
    assert "mean" in d.index and "50%" in d.index
    assert round(float(d.loc["mean", "maths"]), 2) == 74.00

    print(f"  shape {df.shape}, dtypes {df.dtypes.nunique()} distinct, "
          f"dept counts {df.dept.value_counts().to_dict()}")
    print(f"       df.info() is the FIRST thing to run: dtypes and non-null "
          f"counts together")


def index_objects():
    df = students()
    d2 = df.set_index("name")
    assert d2.index.name == "name"
    assert d2.loc["Asha", "maths"] == 88
    # reset_index puts the label back as the FIRST column
    assert list(d2.reset_index().columns)[0] == "name"

    # The Index is IMMUTABLE
    try:
        df.index[0] = "x"
        raise AssertionError("expected a TypeError")
    except TypeError:
        pass

    # ...but replacing the whole index is fine
    d3 = df.copy()
    d3.index = list("abcde")
    assert d3.index.tolist() == list("abcde")

    assert "mathematics" in df.rename(columns={"maths": "mathematics"}).columns

    print("  Index is immutable (set_index/reset_index/rename replace it wholesale)")


def new_columns():
    df = students()
    df["total"] = df.maths + df.stats
    assert df.total.tolist() == [179, 123, 183, 137, 99]

    df["avg"] = df[["maths", "stats"]].mean(axis=1)
    assert df.avg.iloc[0] == 89.5

    # Attribute assignment does NOT create a column.
    import warnings
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        df.newcol = 0
    assert "newcol" not in df.columns, "df.newcol = 0 sets an ATTRIBUTE, not a column"
    df["newcol"] = 0
    assert "newcol" in df.columns

    print(f"  totals {df.total.tolist()}; df['x'] = ... creates a column, df.x = ... does not")


def main():
    print("Practical 5 -- Series and DataFrame")
    # Step 1: Create a Series
    series_creation()
    # Step 2: Create a DataFrame
    dataframe_creation()
    # Step 3: Inspect it
    attributes_and_inspection()
    # Step 4: Work with Index objects
    index_objects()
    # Step 5: Add columns
    new_columns()


if __name__ == "__main__":
    main()

5. Execution and Results

Practical 5 -- Series and DataFrame
  Series: 4 labelled values, array-like and dict-like at once
  DataFrame: dict -> columns, list of lists -> rows, list of dicts -> rows
  shape (5, 5), dtypes 2 distinct, dept counts {'DS': 3, 'Stats': 2}
       df.info() is the FIRST thing to run: dtypes and non-null counts together
  Index is immutable (set_index/reset_index/rename replace it wholesale)
  totals [179, 123, 183, 137, 99]; df['x'] = ... creates a column, df.x = ... does not

Practical 6 — Indexing, selection, filtering and boolean indexing

1. Question

Select, filter and index the rows and columns of a DataFrame.

2. Aim

Select with [], loc and iloc, filter with masks and query, and set values safely in Pandas 3.

3. Steps

1. Select with [], loc and iloc.
2. See loc include its end and iloc not.
3. Filter rows.
4. See why and raises.
5. Set values on a selection, under Pandas 3.

sub = df[df.dept == "DS"]
sub["maths"] = 100            # Pandas 3: NO warning, and df is UNCHANGED

4. Programme

"""Practical 6 — Indexing, selection, filtering and boolean indexing."""
import warnings
import numpy as np
import pandas as pd
from fixtures import students


def three_accessors():
    df = students()

    assert isinstance(df["maths"], pd.Series), "one column -> Series"
    assert isinstance(df[["maths", "stats"]], pd.DataFrame), "a list -> DataFrame"

    assert df.loc[0, "maths"] == 88
    assert df.iloc[0, 3] == 88
    assert df.at[0, "maths"] == 88
    assert df.iat[0, 3] == 88
    assert df.iloc[-1]["name"] == "Bhanu"

    print("  [] selects columns; .loc by label; .iloc by position; .at/.iat one cell")


def loc_is_inclusive_iloc_is_not():
    """The asymmetry that looks like a bug and is deliberate."""
    df = students()
    assert len(df.loc[0:2]) == 3, "labels INCLUSIVE"
    assert len(df.iloc[0:2]) == 2, "positions EXCLUSIVE"

    # With labels you often do not know what follows, so an exclusive endpoint
    # would make a column slice unusable.
    cols = df.loc[:, "name":"maths"].columns.tolist()
    assert cols == ["name", "dept", "maths"], "'maths' is INCLUDED"

    print("  .loc[0:2] -> 3 rows; .iloc[0:2] -> 2 rows; labels inclusive, positions not")


def filtering():
    df = students()

    assert df[df.maths > 70].name.tolist() == ["Asha", "Meena", "Kiran"]
    assert df[(df.maths > 70) & (df.dept == "DS")].name.tolist() == ["Asha", "Kiran"]
    assert df[df.dept.isin(["Stats"])].name.tolist() == ["Meena", "Bhanu"]
    assert df[~df.dept.isin(["Stats"])].shape[0] == 3
    assert df[df.maths.between(60, 90)].name.tolist() == ["Asha", "Ravi", "Kiran"]
    assert df[df.name.str.startswith("A")].name.tolist() == ["Asha"]

    # query() gives the same answer, more readably
    a = df[(df.maths > 70) & (df.dept == "DS")]
    b = df.query("maths > 70 and dept == 'DS'")
    assert a.equals(b)

    threshold = 70
    assert df.query("maths > @threshold").shape[0] == 3, "@ refers to a Python variable"

    print("  filters agree between & and query(); between() is inclusive both ends")


def and_raises():
    df = students()
    try:
        df[(df.maths > 70) and (df.dept == "DS")]
        raise AssertionError("expected ValueError")
    except ValueError as e:
        assert "ambiguous" in str(e)
    print("  `and` raises 'truth value is ambiguous' -- use & with parentheses")


def setting_with_copy():
    """Pandas 3 changed this. Know exactly how."""
    df = students()
    before = df.loc[df.dept == "DS", "maths"].tolist()
    assert before == [88, 65, 71]

    sub = df[df.dept == "DS"]
    with warnings.catch_warnings(record=True) as caught:
        warnings.simplefilter("always")
        sub["maths"] = 100
    warned = [w for w in caught if "SettingWithCopy" in w.category.__name__]

    # Pandas 3: copy-on-write means NO warning AND the original is untouched.
    assert df.loc[df.dept == "DS", "maths"].tolist() == before, \
        "the original must be unchanged under copy-on-write"
    assert warned == [], "Pandas 3 no longer warns -- the bug is now SILENT"

    # Correct form 1: modify the original, in ONE .loc
    d1 = students()
    d1.loc[d1.dept == "DS", "maths"] = 100
    assert d1.loc[d1.dept == "DS", "maths"].tolist() == [100, 100, 100]

    # Correct form 2: work separately, and say so
    d2 = students()
    part = d2[d2.dept == "DS"].copy()
    part["maths"] = 100
    assert d2.loc[d2.dept == "DS", "maths"].tolist() == before, "d2 untouched, intentionally"

    print("  Pandas 3: chained assignment leaves the original UNCHANGED and")
    print("       raises NO warning -- it now fails silently and completely")
    print("       fix 1: df.loc[mask, 'col'] = x    fix 2: .copy() first")


def main():
    print("Practical 6 -- Selection and filtering")
    # Step 1: Select with [], loc and iloc
    three_accessors()
    # Step 2: See loc include its end and iloc not
    loc_is_inclusive_iloc_is_not()
    # Step 3: Filter rows
    filtering()
    # Step 4: See why `and` raises
    and_raises()
    # Step 5: Set values on a selection, under Pandas 3
    setting_with_copy()


if __name__ == "__main__":
    main()

5. Execution and Results

Practical 6 -- Selection and filtering
  [] selects columns; .loc by label; .iloc by position; .at/.iat one cell
  .loc[0:2] -> 3 rows; .iloc[0:2] -> 2 rows; labels inclusive, positions not
  filters agree between & and query(); between() is inclusive both ends
  `and` raises 'truth value is ambiguous' -- use & with parentheses
  Pandas 3: chained assignment leaves the original UNCHANGED and
       raises NO warning -- it now fails silently and completely
       fix 1: df.loc[mask, 'col'] = x    fix 2: .copy() first

Practical 7 — Arithmetic and data alignment

1. Question

Perform arithmetic between Series and DataFrames, and see how Pandas aligns them.

2. Aim

Add and subtract labelled data, and see alignment by label at work.

3. Steps

1. Align arithmetic by label.
2. Fill the gaps with fill_value.
3. Combine a DataFrame and a Series.
4. Align two DataFrames.

a = pd.Series([10,20,30], index=["x","y","z"])
b = pd.Series([1,2,3,4],  index=["w","x","y","z"])
a + b        # w NaN, x 12, y 23, z 34

4. Programme

"""Practical 7 — Arithmetic and data alignment between Series and DataFrames.

Alignment is the feature that most distinguishes Pandas from NumPy, and the
one that turns a whole class of silent bugs into visible NaNs.
"""
import numpy as np
import pandas as pd
from fixtures import students


def alignment_is_by_label():
    a = pd.Series([10, 20, 30], index=["x", "y", "z"])
    b = pd.Series([1, 2, 3, 4], index=["w", "x", "y", "z"])

    r = a + b
    assert r.index.tolist() == ["w", "x", "y", "z"], "the UNION of both indexes"
    assert np.isnan(r["w"]), "w is in b only"
    assert r["x"] == 12 and r["y"] == 23 and r["z"] == 34
    assert r.dtype == np.float64, "NaN is a float, so the result is float64"

    # The decisive point: x sits at position 0 in a and position 1 in b, yet
    # the answer is right, because Pandas matched LABEL to LABEL.
    assert a.index.get_loc("x") == 0
    assert b.index.get_loc("x") == 1

    # NumPy, with no labels, would have added mismatched pairs -- and produced
    # a plausible, wrong answer with no warning at all.
    common = ["x", "y", "z"]
    positional = a.to_numpy() + b.to_numpy()[:3]
    assert positional.tolist() == [11, 22, 33], "wrong, and silently so"
    assert a.add(b)[common].tolist() == [12.0, 23.0, 34.0], "right"

    print(f"  a + b -> {dict(zip(r.index, [None if pd.isna(v) else v for v in r]))}")
    print(f"       positional would give {positional.tolist()} -- plausible and WRONG")


def fill_value():
    a = pd.Series([10, 20, 30], index=["x", "y", "z"])
    b = pd.Series([1, 2, 3, 4], index=["w", "x", "y", "z"])

    r = a.add(b, fill_value=0)
    assert r["w"] == 1, "0 + 1"
    assert r["x"] == 12 and r["y"] == 23 and r["z"] == 34
    assert not r.isna().any()

    assert a.sub(b, fill_value=0)["w"] == -1
    assert a.mul(b, fill_value=1)["w"] == 1

    print(f"  fill_value=0 -> w = {r['w']}, and no NaN remains")


def dataframe_series_arithmetic():
    df = students().set_index("name")[["maths", "stats"]]

    doubled = df * 2
    assert doubled.loc["Asha", "maths"] == 176

    # By DEFAULT a DataFrame-Series operation matches the Series' index against
    # the DataFrame's COLUMNS and broadcasts down the rows.
    centred_cols = df - df.mean()
    assert centred_cols.shape == df.shape
    assert abs(centred_cols.maths.mean()) < 1e-12, "each column now has mean 0"

    # To match against the INDEX instead, pass axis=0.
    centred_rows = df.sub(df.mean(axis=1), axis=0)
    assert centred_rows.shape == df.shape
    assert abs(centred_rows.loc["Asha"].mean()) < 1e-12, "each ROW now has mean 0"

    print("  df - df.mean() centres COLUMNS; df.sub(df.mean(axis=1), axis=0) centres ROWS")


def alignment_across_dataframes():
    a = pd.DataFrame({"x": [1, 2], "y": [3, 4]}, index=["p", "q"])
    b = pd.DataFrame({"y": [10, 20], "z": [30, 40]}, index=["q", "r"])

    r = a + b
    assert sorted(r.columns) == ["x", "y", "z"], "union of COLUMNS"
    assert sorted(r.index) == ["p", "q", "r"], "union of INDEX"
    assert r.loc["q", "y"] == 14, "the only cell present in both"
    assert r.isna().sum().sum() == 8, "every other cell is NaN"

    print(f"  DataFrame + DataFrame aligns on BOTH axes: only ('q','y') overlaps")


def main():
    print("Practical 7 -- Arithmetic and data alignment")
    # Step 1: Align arithmetic by label
    alignment_is_by_label()
    # Step 2: Fill the gaps with fill_value
    fill_value()
    # Step 3: Combine a DataFrame and a Series
    dataframe_series_arithmetic()
    # Step 4: Align two DataFrames
    alignment_across_dataframes()


if __name__ == "__main__":
    main()

5. Execution and Results

Practical 7 -- Arithmetic and data alignment
  a + b -> {'w': None, 'x': 12.0, 'y': 23.0, 'z': 34.0}
       positional would give [11, 22, 33] -- plausible and WRONG
  fill_value=0 -> w = 1.0, and no NaN remains
  df - df.mean() centres COLUMNS; df.sub(df.mean(axis=1), axis=0) centres ROWS
  DataFrame + DataFrame aligns on BOTH axes: only ('q','y') overlaps

Practical 8 — Sorting, ranking, dropping and duplicate indexes

1. Question

Sort and rank data, drop entries, and handle duplicate index labels.

2. Aim

Sort and rank, with each way of breaking ties, and see what duplicate labels do.

3. Steps

1. Sort.
2. Rank, and break ties.
3. Drop rows and columns.
4. Handle duplicate index labels.
5. Find duplicate rows.

4. Programme

"""Practical 8 — Sorting, ranking, dropping entries and duplicate indexes."""
import numpy as np
import pandas as pd
from fixtures import students


def sorting():
    df = students()
    assert df.sort_values("maths").name.tolist() == \
        ["Bhanu", "Ravi", "Kiran", "Asha", "Meena"]
    assert df.sort_values("maths", ascending=False).name.iloc[0] == "Meena"

    multi = df.sort_values(["dept", "maths"], ascending=[True, False])
    assert multi.name.tolist() == ["Asha", "Kiran", "Ravi", "Meena", "Bhanu"]

    assert df.nlargest(3, "maths").name.tolist() == ["Meena", "Asha", "Kiran"]
    assert df.nsmallest(2, "maths").name.tolist() == ["Bhanu", "Ravi"]

    # NaN placement
    d = df.copy()
    d.loc[0, "maths"] = np.nan
    assert pd.isna(d.sort_values("maths").maths.iloc[-1]), "NaN sorts LAST by default"
    assert pd.isna(d.sort_values("maths", na_position="first").maths.iloc[0])

    print("  sort_values, multi-key with per-key direction, nlargest, NaN placement")


def ranking():
    """The tie-breaking methods -- the whole examinable point."""
    s = pd.Series([70, 85, 70, 92, 60])

    expected = {
        "average": [2.5, 4.0, 2.5, 5.0, 1.0],
        "min":     [2.0, 4.0, 2.0, 5.0, 1.0],
        "max":     [3.0, 4.0, 3.0, 5.0, 1.0],
        "first":   [2.0, 4.0, 3.0, 5.0, 1.0],
        "dense":   [2.0, 3.0, 2.0, 4.0, 1.0],
    }
    for method, want in expected.items():
        got = s.rank(method=method).tolist()
        assert got == want, f"{method}: {got} != {want}"

    assert s.rank().tolist() == expected["average"], "average is the DEFAULT"
    assert s.rank(ascending=False).tolist() == [3.5, 2.0, 3.5, 1.0, 5.0]
    assert s.rank(pct=True).max() == 1.0

    # min leaves a GAP after ties; dense does not.
    assert 3.0 not in expected["min"], "min: 1, 2, 2, 4 -- a gap at 3"
    assert 3.0 in expected["dense"], "dense: 1, 2, 2, 3 -- no gap"

    print("  ranking [70,85,70,92,60], the two 70s get:")
    for method, want in expected.items():
        print(f"    {method:8s} {want}")


def dropping():
    df = students()

    assert len(df.drop(0)) == 4
    assert len(df.drop([0, 2])) == 3
    assert "maths" not in df.drop(columns="maths").columns
    assert "maths" not in df.drop("maths", axis=1).columns

    # drop returns a COPY -- df itself is unchanged
    df.drop(columns="maths")
    assert "maths" in df.columns, "drop does NOT modify in place"

    d = df.copy()
    d.loc[0, "maths"] = np.nan
    d.loc[1, ["maths", "stats"]] = np.nan
    assert len(d.dropna()) == 3, "any NaN in the row removes it"
    assert len(d.dropna(subset=["stats"])) == 4
    # Row 0 has 4 non-null values and row 1 has 3; the rest have all 5.
    assert d.notna().sum(axis=1).tolist() == [4, 3, 5, 5, 5]
    assert len(d.dropna(thresh=4)) == 4, "keep rows with >= 4 non-null values"
    assert len(d.dropna(thresh=5)) == 3, "keep only complete rows"

    print("  drop returns a COPY; dropna(subset=) and thresh= narrow the criterion")


def duplicate_indexes():
    """The return TYPE depends on the data, which is why code breaks later."""
    s = pd.Series([1, 2, 3, 4], index=["a", "a", "b", "c"])

    assert s.index.is_unique is False
    assert isinstance(s["a"], pd.Series) and len(s["a"]) == 2
    assert np.isscalar(s["b"]) or isinstance(s["b"], (int, np.integer))

    assert s.index.duplicated().tolist() == [False, True, False, False]
    assert len(s[~s.index.duplicated(keep="first")]) == 3

    print("  duplicate index: s['a'] -> a SERIES, s['b'] -> a SCALAR")
    print("       code written against a unique index breaks when one appears")


def duplicate_rows():
    df = pd.DataFrame({
        "roll": [21, 22, 21, 23, 22],
        "name": ["Asha", "Ravi", "Asha", "Meena", "Ravi Teja"],
        "marks": [88, 65, 88, 94, 65],
    })

    assert df.duplicated().sum() == 1, "only row 2 is a FULL duplicate"
    assert df.duplicated(subset=["roll"]).sum() == 2, "two rolls repeat"
    assert df.duplicated(subset=["roll"], keep=False).sum() == 4, \
        "keep=False marks ALL members of every duplicate group"

    assert len(df.drop_duplicates()) == 4
    assert len(df.drop_duplicates(subset=["roll"])) == 3, "one row per student"
    assert df.drop_duplicates(subset=["roll"], keep="last").name.tolist() == \
        ["Asha", "Meena", "Ravi Teja"]

    print("  duplicated(): 1 full duplicate, but 2 duplicated ROLLS --")
    print("       'Ravi' and 'Ravi Teja' are the same student with a typo, and")
    print("       a bare drop_duplicates() would miss it. Specify the subset.")


def main():
    print("Practical 8 -- Sorting, ranking, dropping, duplicates")
    # Step 1: Sort
    sorting()
    # Step 2: Rank, and break ties
    ranking()
    # Step 3: Drop rows and columns
    dropping()
    # Step 4: Handle duplicate index labels
    duplicate_indexes()
    # Step 5: Find duplicate rows
    duplicate_rows()


if __name__ == "__main__":
    main()