Skip to the content

Software / Tools Used

Exercise 1: Simple Linear Regression by OLS

1. Problem

For \(X = (10, 20, 30, 40, 50)\) and \(Y = (11, 20, 27, 34, 43)\), fit the regression of \(Y\) on \(X\) by ordinary least squares, and find the residuals and the estimate of \(\sigma^2\).

2. Aim

To estimate the intercept and slope of a simple linear regression by least squares, and the error variance from the residuals.

3. Formula

\[ \hat\beta_1 = \frac{S_{XY}}{S_{XX}}, \quad \hat\beta_0 = \bar Y - \hat\beta_1 \bar X, \quad \hat\sigma^2 = \frac{\text{RSS}}{n-2} \]

Applying it:

  1. Find \(\bar X, \bar Y\), \(S_{XY} = \sum (X-\bar X)(Y-\bar Y)\) and \(S_{XX} = \sum (X-\bar X)^2\).
  2. Compute \(\hat\beta_1\), then \(\hat\beta_0\).
  3. Compute the fitted values and residuals \(\hat u = Y - \hat Y\); RSS \(= \sum \hat u^2\).

4. Calculation

\(\bar X = 30, \bar Y = 27, S_{XY} = 780, S_{XX} = 1000\).

\(\hat\beta_1 = 780/1000 = 0.78\), \(\hat\beta_0 = 27 - 0.78 \times 30 = 3.60\).

\(\hat Y = 3.60 + 0.78 X\). Residuals \(-0.4, 0.8, 0, -0.8, 0.4\); RSS = 1.6, \(\hat\sigma^2 = 1.6/3 = 0.533\).

CHECK IN R 
x <- c(10,20,30,40,50); y <- c(11,20,27,34,43)
fit <- lm(y ~ x)
coef(fit)
round(resid(fit), 2)
(Intercept)           x 
       3.60        0.78 
   1    2    3    4    5 
-0.4  0.8  0.0 -0.8  0.4

5. Result

The fitted line is \(\hat Y = 3.60 + 0.78X\): each extra unit of \(X\) raises \(Y\) by 0.78 on average. RSS = 1.6 and \(\hat\sigma^2 = 0.533\) on 3 degrees of freedom.

Exercise 2: Computing R² and the F-Statistic

1. Problem

For the regression of Exercise 1, find \(R^2\) and the \(F\) statistic for \(H_0: \beta_1 = 0\), and test it at 5%.

2. Aim

To measure the share of the variation in Y explained by the regression, and to test the regression as a whole.

3. Formula

\[ \text{TSS} = \text{ESS} + \text{RSS}, \quad R^2 = \frac{\text{ESS}}{\text{TSS}}, \quad F = \frac{\text{ESS}/1}{\text{RSS}/(n-2)} \sim F_{1,\,n-2} \]

Applying it:

  1. TSS \(= \sum (Y-\bar Y)^2\); ESS = TSS − RSS.
  2. \(R^2 = \text{ESS}/\text{TSS}\); \(F = \text{ESS}/\hat\sigma^2\).
  3. Compare \(F\) with \(F_{0.05,\,1,\,3}\).

4. Calculation

Continuing Exercise 1: TSS = \(\sum(Y - 27)^2 = 610\), RSS = 1.6, ESS = 608.4.

\(R^2 = 608.4/610 = 0.9974\). \(F = 608.4/(1.6/3) = 608.4/0.533 = 1140.8\) with df = (1, 3).

Critical \(F_{0.05, 1, 3} = 10.13\).

CHECK IN R 
anova(fit)
          Df Sum Sq Mean Sq F value    Pr(>F)    
x          1  608.4  608.40  1140.8 5.706e-05 ***
Residuals  3    1.6    0.53

5. Result

\(R^2 = 0.9974\): the regression explains 99.7% of the variation in \(Y\). \(F = 1140.8 > 10.13\), so \(H_0: \beta_1 = 0\) is rejected overwhelmingly.

Exercise 3: Multiple Regression via Matrix Algebra

1. Problem

For \(Y\) on \(X_1, X_2\) with three observations, \(\mathbf X = \begin{pmatrix}1&1&2\\1&2&1\\1&3&2\end{pmatrix}\) and \(\mathbf Y = \begin{pmatrix}4\\5\\7\end{pmatrix}\), find the least-squares estimates by matrix algebra.

2. Aim

To compute the least-squares estimates of a multiple regression from the normal equations in matrix form.

3. Formula

\[ \hat{\boldsymbol\beta} = (\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf Y \]

Applying it:

  1. Form \(\mathbf X'\mathbf X\) and \(\mathbf X'\mathbf Y\).
  2. Solve \((\mathbf X'\mathbf X)\hat{\boldsymbol\beta} = \mathbf X'\mathbf Y\).

4. Calculation

\(\mathbf X'\mathbf X = \begin{pmatrix}3&6&5\\6&14&10\\5&10&9\end{pmatrix}, \mathbf X'\mathbf Y = \begin{pmatrix}16\\35\\27\end{pmatrix}.\)

\(\hat{\boldsymbol\beta} = (\mathbf X'\mathbf X)^{-1}\mathbf X'\mathbf Y = (1.5,\ 1.5,\ 0.5)'\).

CHECK IN R 
X <- cbind(1, c(1, 2, 3), c(2, 1, 2)); Y <- c(4, 5, 7)
solve(t(X) %*% X, t(X) %*% Y)
     [,1]
[1,]  1.5
[2,]  1.5
[3,]  0.5

5. Result

\(\hat Y = 1.5 + 1.5X_1 + 0.5X_2\). With three observations and three parameters the fit is exact (every residual is 0 and there are no degrees of freedom left to estimate \(\sigma^2\)); a real regression needs \(n > k\).

Exercise 4: Confidence Interval & t-Test

1. Problem

For the slope of Exercise 1 (\(\hat\beta_1 = 0.78\), \(\hat\sigma^2 = 0.533\), \(S_{XX} = 1000\), \(n = 5\)), test \(H_0: \beta_1 = 0\) at 5% and find a 95% confidence interval.

2. Aim

To test a single regression coefficient with the t statistic and to give its confidence interval.

3. Formula

\[ \mathrm{se}(\hat\beta_1) = \sqrt{\hat\sigma^2/S_{XX}}, \quad t = \frac{\hat\beta_1}{\mathrm{se}(\hat\beta_1)} \sim t_{n-2}, \quad \hat\beta_1 \pm t_{0.025,\,n-2}\,\mathrm{se}(\hat\beta_1) \]

Applying it:

  1. Compute the standard error and \(t\).
  2. Compare \(|t|\) with \(t_{0.025,\,3}\).
  3. Form the interval.

4. Calculation

\(\mathrm{se}(\hat\beta_1) = \sqrt{0.533/1000} = 0.0231\), df = 3.

\(t = 0.78/0.0231 = 33.8\) (and \(t^2 = 1141 = F\) from Exercise 2). \(t_{0.025, 3} = 3.182\).

95% CI: \(0.78 \pm 3.182 \times 0.0231 = (0.707,\ 0.853)\).

CHECK IN R 
confint(fit)["x", ]
 2.5 % 97.5 % 
0.7065 0.8535

5. Result

\(t = 33.8 > 3.182\): reject \(H_0\); the slope is significant. With 95% confidence \(\beta_1\) lies between 0.707 and 0.853.

Exercise 5: Detecting Heteroscedasticity — Breusch–Pagan

1. Problem

A simple regression on \(n = 50\) observations is fitted. Regressing \(p_i = \hat u_i^2/\hat\sigma^2\) on \(X\) gives ESS\(_p = 13.4\). Test for heteroscedasticity at 5%.

2. Aim

To test whether the error variance changes with the regressor, by the Breusch–Pagan test.

3. Formula

\[ p_i = \frac{\hat u_i^2}{\hat\sigma^2}, \quad \text{BP} = \tfrac12\,\text{ESS}_p \sim \chi^2_{k} \text{ under } H_0 \]

Applying it:

  1. Fit OLS and get \(\hat u\).
  2. Compute \(p_i = \hat u_i^2 /\hat\sigma^2\).
  3. Regress \(p_i\) on \(X\); get ESS\(_p\).
  4. BP \(= \tfrac{1}{2}\text{ESS}_p\); compare with \(\chi^2_{0.05}\) on as many df as regressors.

4. Calculation

BP = 13.4/2 = 6.7, df = 1 (one regressor). \(\chi^2_{0.05, 1} = 3.84\).

IN R, WITH YOUR OWN DATA 
library(lmtest); bptest(fit)

The data of this exercise are given only as summaries, so the test is worked by hand; with the data, this line gives the statistic and its p-value (the lmtest package).

5. Result

Since \(6.7 > 3.84\), reject \(H_0\) of homoscedasticity: the error variance is not constant.

Exercise 6: White's Test for Heteroscedasticity

1. Problem

\(Y\) is regressed on \(X_1, X_2\) (\(n = 60\)). The auxiliary regression of \(\hat u^2\) on \(X_1, X_2, X_1^2, X_2^2, X_1 X_2\) has \(R^2 = 0.18\). Apply White's test at 5%.

2. Aim

To test for heteroscedasticity of an unknown form, by White's general test.

3. Formula

\[ W = nR^2_{\text{aux}} \sim \chi^2_{q}, \quad q = \text{number of regressors in the auxiliary regression} \]

Applying it:

  1. Fit OLS; get \(\hat u\).
  2. Regress \(\hat u^2\) on the regressors, their squares and cross-product; note \(R^2\).
  3. \(W = nR^2\); compare with \(\chi^2_{0.05,\,5}\).

4. Calculation

White statistic \(= nR^2 = 60 \times 0.18 = 10.8\), df = 5. \(\chi^2_{0.05, 5} = 11.07\).

IN R, WITH YOUR OWN DATA 
library(lmtest); bptest(fit, ~ X1 + X2 + I(X1^2) + I(X2^2) + I(X1*X2))

White's test is the Breusch–Pagan test with the squares and cross-product added.

5. Result

Since \(10.8 < 11.07\), do not reject \(H_0\): homoscedasticity is accepted, though only just.

Exercise 7: Computing VIF for Multicollinearity

1. Problem

A regression has three regressors. Regressing each on the other two gives \(R_1^2 = 0.85\), \(R_2^2 = 0.30\) and \(R_3^2 = 0.78\). Compute the variance inflation factors and judge the multicollinearity.

2. Aim

To measure how far each regressor is explained by the others, by its variance inflation factor and tolerance.

3. Formula

\[ \text{VIF}_j = \frac{1}{1 - R_j^2}, \quad \text{TOL}_j = 1 - R_j^2 \]

Applying it:

  1. For each regressor, take \(R_j^2\) from its regression on the others.
  2. Compute VIF\(_j\) and TOL\(_j\).
  3. Judge: VIF above 5 is a warning, above 10 severe.

4. Calculation

VIF\(_1\) = \(1/(1 - 0.85) = 6.67\), TOL\(_1\) = 0.15.

VIF\(_2\) = \(1/(1 - 0.30) = 1.43\); VIF\(_3\) = \(1/(1 - 0.78) = 4.55\).

IN R, WITH YOUR OWN DATA 
library(car); vif(fit)

With the data, this gives every VIF at once (the car package).

5. Result

VIF\(_1\) = 6.67 is borderline severe (5–10). Consider dropping \(X_1\) or replacing it with a function of \(X_2\) and \(X_3\).

Exercise 8: Durbin–Watson Test

1. Problem

Residuals from a regression on \(n = 30\) time-series observations, with two regressors, give \(\sum (\hat u_t - \hat u_{t-1})^2 = 5.5\) and \(\sum \hat u_t^2 = 7.0\). Test for first-order autocorrelation at 5%.

2. Aim

To test for first-order autocorrelation of the errors by the Durbin–Watson statistic.

3. Formula

\[ d = \frac{\sum_{t=2}^{n}(\hat u_t - \hat u_{t-1})^2}{\sum_{t=1}^{n}\hat u_t^2} \approx 2(1 - \hat\rho) \]

Applying it:

  1. Compute \(d\) and \(\hat\rho \approx 1 - d/2\).
  2. Read \(d_L, d_U\) for \(n\) and \(k\) from the Durbin–Watson table.
  3. \(d < d_L\): positive autocorrelation; \(d > d_U\): none; between: inconclusive.

4. Calculation

\(d = 5.5/7.0 = 0.786\). Implied \(\hat\rho \approx 1 - 0.786/2 = 0.607\).

From DW tables (\(n = 30, k = 2\)): \(d_L = 1.28, d_U = 1.57\).

IN R, WITH YOUR OWN DATA 
library(lmtest); dwtest(fit)

With the data, this gives d and its p-value.

5. Result

\(0.786 < 1.28\), so reject \(H_0\): there is strong positive autocorrelation (\(\hat\rho \approx 0.61\)).

Exercise 9: Cochrane–Orcutt Iterative Procedure

1. Problem

The regression of Exercise 8 has autocorrelated errors with \(\hat\rho = 0.6\). Remove the autocorrelation by the Cochrane–Orcutt iterative method.

2. Aim

To estimate a regression with AR(1) errors by transforming the data with the estimated autocorrelation, repeatedly, until it settles.

3. Formula

\[ Y_t^* = Y_t - \hat\rho Y_{t-1}, \quad X_t^* = X_t - \hat\rho X_{t-1}, \quad \hat\rho = \frac{\sum \hat u_t \hat u_{t-1}}{\sum \hat u_{t-1}^2} \]

Applying it:

  1. Start from the OLS residuals; estimate \(\hat\rho\).
  2. Transform \(Y\) and \(X\) with \(\hat\rho\); fit OLS to the transformed data.
  3. From the new residuals re-estimate \(\hat\rho\); repeat until it changes little.

4. Calculation

Initial OLS gives \(\hat\rho = 0.6\) (from Exercise 8). Transform: \(Y_t^* = Y_t - 0.6 Y_{t-1}\), \(X_t^* = X_t - 0.6 X_{t-1}\).

Run OLS on the transformed model. Recompute residuals; get new \(\hat\rho^{(1)} = 0.18\). Re-transform with 0.18. Continue.

IN R, WITH YOUR OWN DATA 
library(orcutt); cochrane.orcutt(fit)

With the data, this runs the iterations (the orcutt package).

5. Result

The procedure typically converges in about 3 iterations, here to \(\hat\rho \approx 0.15\), with the new DW close to 2: the remaining autocorrelation is negligible.

Exercise 10: Robust Standard Errors

1. Problem

OLS reports \(\hat\beta_1 = 0.50\) with \(\mathrm{se}_{\text{OLS}} = 0.10\). The heteroscedasticity-robust standard error is 0.18 and the Newey–West (HAC) standard error is 0.22. Recompute the t statistics.

2. Aim

To see how heteroscedasticity- and autocorrelation-robust standard errors change the test of a coefficient.

3. Formula

\[ t = \frac{\hat\beta_1}{\mathrm{se}}, \quad \mathrm{se}_{\text{HC}} = \sqrt{\big[(\mathbf X'\mathbf X)^{-1}\mathbf X'\hat{\boldsymbol\Omega}\mathbf X(\mathbf X'\mathbf X)^{-1}\big]_{jj}} \]

Applying it:

  1. Compute \(t\) with each standard error.
  2. Compare each with the critical value.

4. Calculation

OLS: \(t = 0.50/0.10 = 5.0\). Robust (HC): \(t = 0.50/0.18 = 2.78\). HAC (Newey–West): \(t = 0.50/0.22 = 2.27\).

IN R, WITH YOUR OWN DATA 
library(sandwich); library(lmtest)
coeftest(fit, vcov = vcovHC(fit, "HC1"))   # heteroscedasticity-robust
coeftest(fit, vcov = NeweyWest(fit))       # HAC

With the data, these give the coefficient table with each kind of standard error.

5. Result

The coefficient stays significant, but the margin is much smaller with robust (\(t = 2.78\)) and HAC (\(t = 2.27\)) standard errors. Always report robust / HAC SEs alongside the OLS SEs.

← Previous: Unit 5 Next: Syllabus →