Unit 1 — Analysis of Variance (ANOVA)

\[ \underbrace{\sum (y_{ij} - \bar y)^2}_{\text{Total SS}} = \underbrace{\sum n_i (\bar y_i - \bar y)^2}_{\text{Between (treatment) SS}} + \underbrace{\sum (y_{ij} - \bar y_i)^2}_{\text{Within (error) SS}} \]

STATEMENT

Let \(X_1, \ldots, X_n\) be independent \(N(0, \sigma^2)\), and suppose

\[ \sum_{i=1}^n X_i^2 = Q_1 + Q_2 + \cdots + Q_k , \]

where each \(Q_j\) is a quadratic form in the \(X_i\) with rank \(r_j\). Then \(Q_1/\sigma^2, \ldots, Q_k/\sigma^2\) are independent \(\chi^2\) variates with \(r_1, \ldots, r_k\) degrees of freedom if and only if \(r_1 + r_2 + \cdots + r_k = n\).

ANOVA uses it directly: the total sum of squares is split into pieces whose degrees of freedom add up to the total, so the pieces are independent chi-squares, and their mean squares can be compared by an F-ratio. (The textbook's sum runs to \(k\) on the left; it runs over all \(n\) variables.)

MATRIX FORM

For \(n\) observations and \(m\) unknown parameters,

\[ y = A\beta + \varepsilon, \qquad y = \begin{bmatrix} y_1 \\ \vdots \\ y_n \end{bmatrix},\; A = \begin{bmatrix} a_{11} & \cdots & a_{1m} \\ \vdots & & \vdots \\ a_{n1} & \cdots & a_{nm} \end{bmatrix},\; \beta = \begin{bmatrix} \beta_1 \\ \vdots \\ \beta_m \end{bmatrix},\; \varepsilon = \begin{bmatrix} \varepsilon_1 \\ \vdots \\ \varepsilon_n \end{bmatrix}, \]

with \(A\) a known \(n \times m\) matrix of coefficients, and

1. \(E(\varepsilon) = 0\), i.e. \(E(\varepsilon_i) = 0\) for every \(i\);
2. \(E(\varepsilon\varepsilon') = \sigma^2 I\): each \(\varepsilon_i\) has variance \(\sigma^2\), and different errors are uncorrelated;
3. for the tests, \(\varepsilon \sim N(0, \sigma^2 I)\), so \(y \sim N(A\beta, \sigma^2 I)\).

Then \(E(y_i) = a_{i1}\beta_1 + \cdots + a_{im}\beta_m\), \(\text{Var}(y_i) = \sigma^2\) and \(\text{Cov}(y_i, y_j) = 0\) for \(i \ne j\); \(D(y) = \sigma^2 I\) is the dispersion matrix of \(y\). Assumptions 1 and 2 are enough for least squares to give the best linear unbiased estimates; normality is added for the F-tests.

\[ y_{ij} = \mu + \alpha_i + \epsilon_{ij}, \quad i = 1, \ldots, k;\; j = 1, \ldots, n_i, \]

with \(\sum_i n_i \alpha_i = 0\) and \(\epsilon_{ij} \sim N(0, \sigma^2)\) independently.

\[ \text{SS}_T = \sum_{ij} y_{ij}^2 - C \] \[ \text{SS}_{Tr} = \sum_i \dfrac{T_i^2}{n_i} - C \] \[ \text{SS}_E = \text{SS}_T - \text{SS}_{Tr} \]

NORMAL EQUATIONS

Here \(\mu_i = \mu + \alpha_i\) is the mean of treatment \(i\), \(\mu = \frac1N\sum n_i\mu_i\), and \(\alpha_i = \mu_i - \mu\), so \(\sum n_i\alpha_i = \sum n_i\mu_i - N\mu = 0\). Minimise the error sum of squares \(E = \sum_i\sum_j (y_{ij} - \mu - \alpha_i)^2\):

\[ \frac{\partial E}{\partial \mu} = -2\sum_i\sum_j (y_{ij} - \mu - \alpha_i) = 0 \;\Longrightarrow\; N\bar y_{..} - N\mu - \sum n_i\alpha_i = 0 \;\Longrightarrow\; \hat\mu = \bar y_{..}, \] \[ \frac{\partial E}{\partial \alpha_i} = -2\sum_j (y_{ij} - \mu - \alpha_i) = 0 \;\Longrightarrow\; n_i\bar y_{i.} - n_i\mu - n_i\alpha_i = 0 \;\Longrightarrow\; \hat\alpha_i = \bar y_{i.} - \bar y_{..} . \]

The residual is \(\hat\varepsilon_{ij} = y_{ij} - \hat\mu - \hat\alpha_i = y_{ij} - \bar y_{i.}\).

PROOF OF THE IDENTITY

Write each observation as estimate plus residual: \(y_{ij} - \bar y_{..} = (\bar y_{i.} - \bar y_{..}) + (y_{ij} - \bar y_{i.})\). Squaring and summing over \(i\) and \(j\),

\[ \sum_i\sum_j (y_{ij} - \bar y_{..})^2 = \sum_i n_i(\bar y_{i.} - \bar y_{..})^2 + \sum_i\sum_j (y_{ij} - \bar y_{i.})^2 \] \[ + \; 2\sum_i (\bar y_{i.} - \bar y_{..})\sum_j (y_{ij} - \bar y_{i.}) . \]

The last term is zero because \(\sum_j (y_{ij} - \bar y_{i.}) = n_i\bar y_{i.} - n_i\bar y_{i.} = 0\). So TSS = SST + SSE.

Degrees of freedom. TSS has \(N - 1\) (the \(N\) deviations satisfy \(\sum\sum (y_{ij} - \bar y_{..}) = 0\)); SST has \(k - 1\) (the \(k\) deviations satisfy \(\sum n_i(\bar y_{i.} - \bar y_{..}) = 0\)); SSE has \(N - k\) (its \(N\) deviations satisfy one constraint \(\sum_j (y_{ij} - \bar y_{i.}) = 0\) in each of the \(k\) groups). The degrees of freedom add up as the sums of squares do: \((k - 1) + (N - k) = N - 1\).

Shortcut formulas. Expanding the squares gives TSS \(= \sum\sum y_{ij}^2 - G^2/N\) and SST \(= \sum T_i^2/n_i - G^2/N\), where \(G^2/N\) is the correction factor.

WHY THE F-RATIO WORKS

Averaging the model gives \(\bar y_{i.} = \mu + \alpha_i + \bar\varepsilon_{i.}\) and \(\bar y_{..} = \mu + \bar\varepsilon_{..}\) (using \(\sum n_i\alpha_i = 0\)), with \(E(\bar\varepsilon_{i.}^2) = \sigma^2/n_i\) and \(E(\bar\varepsilon_{..}^2) = \sigma^2/N\). Then

\[ \text{SST} = \sum n_i\big[\alpha_i + (\bar\varepsilon_{i.} - \bar\varepsilon_{..})\big]^2, \qquad \sum n_i(\bar\varepsilon_{i.} - \bar\varepsilon_{..})^2 = \sum n_i\bar\varepsilon_{i.}^2 - N\bar\varepsilon_{..}^2 , \]

and the cross term has expectation zero, so

\[ E(\text{SST}) = \sum n_i\alpha_i^2 + \sum n_i\frac{\sigma^2}{n_i} - N\frac{\sigma^2}{N} = \sum n_i\alpha_i^2 + (k-1)\sigma^2 . \]

Similarly \(\text{SSE} = \sum\sum (\varepsilon_{ij} - \bar\varepsilon_{i.})^2\) \(= \sum\sum\varepsilon_{ij}^2 - \sum n_i\bar\varepsilon_{i.}^2\), so \(E(\text{SSE}) = N\sigma^2 - k\sigma^2 = (N-k)\sigma^2\). Dividing by the degrees of freedom,

\[ E(\text{MS}_{Tr}) = \sigma^2 + \frac{1}{k-1}\sum n_i\alpha_i^2, \qquad E(\text{MS}_E) = \sigma^2 . \]

The error mean square estimates \(\sigma^2\) whatever the treatments do; the treatment mean square estimates \(\sigma^2\) only when \(H_0\) holds, and something larger otherwise. Under \(H_0\), by Cochran's theorem, \(\text{SST}/\sigma^2\) and \(\text{SSE}/\sigma^2\) are independent \(\chi^2_{k-1}\) and \(\chi^2_{N-k}\), so

\[ F = \frac{\text{SST}/(k-1)}{\text{SSE}/(N-k)} = \frac{\text{MS}_{Tr}}{\text{MS}_E} \sim F_{k-1,\,N-k} . \]

Large values of \(F\) count against \(H_0\), so the test uses the upper tail only. An \(F\) below 1 is never significant, and the ratio is never turned upside down to make it exceed 1.

WHICH PAIRS DIFFER?

When \(H_0\) is rejected, the next question is which treatments differ. Two means differ significantly at level \(\alpha\) when their difference exceeds the critical difference (least significant difference)

\[ \text{CD} = t_{\alpha/2,\,N-k}\,\sqrt{\text{MS}_E\Big(\frac{1}{n_i} + \frac{1}{n_j}\Big)} \;=\; t_{\alpha/2,\,N-k}\,\sqrt{\frac{2\,\text{MS}_E}{n}} \;\text{ when every } n_i = n . \]

This is the two-sample \(t\)-test with the pooled error mean square in place of \(\sigma^2\) (which is why \(t\), on the error degrees of freedom, and not \(z\)); \(t_{\alpha/2}\) is the two-tailed point.

Wording. A non-significant \(F\) means the data show no evidence of a difference; it does not prove the means equal. “Accept \(H_0\)” in the textbook should be read as “do not reject \(H_0\)”.

\[ y_{ij} = \mu + \alpha_i + \beta_j + \epsilon_{ij}, \]

with \(\sum \alpha_i = 0,\; \sum \beta_j = 0,\; \epsilon_{ij} \sim N(0, \sigma^2)\).

\[ \text{SS}_T = \sum y^2 - C, \quad \text{SS}_R = \dfrac{\sum R_i^2}{c} - C, \quad \text{SS}_C = \dfrac{\sum C_j^2}{r} - C, \] \[ \text{SS}_E = \text{SS}_T - \text{SS}_R - \text{SS}_C. \]

ESTIMATES AND THE PARTITION

Write \(k\) treatments (rows, \(i\)) and \(h\) varieties or blocks (columns, \(j\)), \(N = hk\); in the notation above \(r = k\), \(c = h\). With \(\sum\alpha_i = 0\) and \(\sum\beta_j = 0\), minimising \(\sum\sum (y_{ij} - \mu - \alpha_i - \beta_j)^2\) gives, exactly as in one-way,

\[ \hat\mu = \bar y_{..}, \qquad \hat\alpha_i = \bar y_{i.} - \bar y_{..}, \qquad \hat\beta_j = \bar y_{.j} - \bar y_{..}, \]

and residual \(y_{ij} - \bar y_{i.} - \bar y_{.j} + \bar y_{..}\). Then

\[ y_{ij} - \bar y_{..} = (\bar y_{i.} - \bar y_{..}) + (\bar y_{.j} - \bar y_{..}) + (y_{ij} - \bar y_{i.} - \bar y_{.j} + \bar y_{..}), \]

and on squaring and summing, all three cross products vanish (each contains a sum of deviations from a mean over a full row or column):

\[ \underbrace{\sum\sum (y_{ij} - \bar y_{..})^2}_{\text{TSS}} = \underbrace{h\sum_i (\bar y_{i.} - \bar y_{..})^2}_{\text{SST}} + \underbrace{k\sum_j (\bar y_{.j} - \bar y_{..})^2}_{\text{SSV}} \] \[ + \underbrace{\sum\sum (y_{ij} - \bar y_{i.} - \bar y_{.j} + \bar y_{..})^2}_{\text{SSE}} . \]

Degrees of freedom: \(hk - 1\) \(= (k - 1) + (h - 1) + (h-1)(k-1)\), the error taking what is left over.

EXPECTED MEAN SQUARES AND THE TWO F-TESTS

With \(\bar y_{i.} = \mu + \alpha_i + \bar\varepsilon_{i.}\), \(\bar y_{.j} = \mu + \beta_j + \bar\varepsilon_{.j}\), \(\bar y_{..} = \mu + \bar\varepsilon_{..}\) and \(E(\bar\varepsilon_{i.}^2) = \sigma^2/h\), \(E(\bar\varepsilon_{.j}^2) = \sigma^2/k\), \(E(\bar\varepsilon_{..}^2) = \sigma^2/hk\):

\[ E(\text{SST}) = h\sum\alpha_i^2 + (k-1)\sigma^2, \qquad E(\text{SSV}) = k\sum\beta_j^2 + (h-1)\sigma^2 . \]

For the error, the residual equals \(\varepsilon_{ij} - \bar\varepsilon_{i.} - \bar\varepsilon_{.j} + \bar\varepsilon_{..}\); expanding its square and taking expectations term by term,

\[ E(\text{SSE}) = \big(hk + k + h + 1 - 2k - 2h\big)\sigma^2 = (h-1)(k-1)\sigma^2 . \]

So \(\text{MS}_E\) always estimates \(\sigma^2\), and

\[ F_T = \frac{\text{MS}_T}{\text{MS}_E} \sim F_{k-1,\,(h-1)(k-1)} \;\text{ under } H_T: \alpha_i = 0, \] \[ F_V = \frac{\text{MS}_V}{\text{MS}_E} \sim F_{h-1,\,(h-1)(k-1)} \;\text{ under } H_V: \beta_j = 0 . \]

Shortcuts: SST \(= \frac1h\sum T_{i.}^2 - \text{CF}\), SSV \(= \frac1k\sum T_{.j}^2 - \text{CF}\), SSE \(=\) TSS \(-\) SST \(-\) SSV.