Skip to the content

Topics Covered

Concept Linear vs Non-linear Regression Lines Regression Coefficients Angle between Lines Correlation vs Regression Explained & Unexplained
On this page
  1. 1. Concept of Regression
  2. 2. Linear and Non-linear Regression
  3. 3. Two Lines of Regression
  4. 4. Properties of Regression Coefficients
  5. 5. Angle Between Two Regression Lines
  6. 6. Worked Examples
  7. 7. Correlation vs Regression
  8. 8. Explained and Unexplained Variation
  9. Key Take-aways

1. Concept of Regression

DEFINITION

Regression is the statistical method of estimating (predicting) the value of one variable from the value of another. It is the average functional relationship between the variables. The term was coined by Francis Galton in his study of "regression toward mediocrity" in human heights.

If \(Y\) is the dependent (response) and \(X\) is the independent (predictor) variable, then the regression of \(Y\) on \(X\) is the relation \(Y = f(X) + \epsilon\) where \(f(x) = E(Y | X = x)\).

2. Linear and Non-linear Regression

This unit focuses on simple linear regression (one predictor).

3. Two Lines of Regression

For bivariate data we fit two regression lines:

3.1 Line of Regression of \(Y\) on \(X\)

\[ Y - \bar y \;=\; b_{yx}(X - \bar x), \qquad b_{yx} \;=\; \dfrac{\text{Cov}(X, Y)}{\sigma_x^2} \;=\; r \cdot \dfrac{\sigma_y}{\sigma_x}. \]

3.2 Line of Regression of \(X\) on \(Y\)

\[ X - \bar x \;=\; b_{xy}(Y - \bar y), \qquad b_{xy} \;=\; \dfrac{\text{Cov}(X, Y)}{\sigma_y^2} \;=\; r \cdot \dfrac{\sigma_x}{\sigma_y}. \]

Both lines pass through \((\bar x, \bar y)\).

(x̄, ȳ) Y on X X on Y Two Lines of Regression Both pass through (x̄, ȳ); angle ↑ as |r| ↓
Fig 4.1 — Two regression lines and their common point (x̄, ȳ)

4. Properties of Regression Coefficients

  1. Geometric mean of the two regression coefficients = correlation coefficient: \[ r \;=\; \pm \sqrt{b_{yx} \cdot b_{xy}}. \]
  2. Both regression coefficients have the same sign, which is also the sign of \(r\).
  3. If one regression coefficient is > 1, the other must be < 1, because their product = \(r^2 \le 1\).
  4. Arithmetic mean of the two regression coefficients \(\ge |r|\): \(\dfrac{b_{yx} + b_{xy}}{2} \ge |r|\) (AM ≥ GM).
  5. Regression coefficients are independent of change of origin but not of scale.
  6. If \(r = \pm 1\): both lines coincide (perfect linear relationship).
  7. If \(r = 0\): the two lines are perpendicular and parallel to the axes (\(Y = \bar y\) and \(X = \bar x\)).

5. Angle Between Two Regression Lines

\[ \tan \theta \;=\; \dfrac{1 - r^2}{|r|} \cdot \dfrac{\sigma_x \sigma_y}{\sigma_x^2 + \sigma_y^2}. \]

6. Worked Examples

EXAMPLE 1 (Find both regression lines)

For the bivariate data:

x12345
y23546

\(\sum x = 15,\; \sum y = 20,\; \sum xy = 69,\; \sum x^2 = 55,\; \sum y^2 = 90,\; n = 5\).

\(\bar x = 3, \bar y = 4\).

\(b_{yx} = (5 \cdot 69 - 15 \cdot 20)/(5 \cdot 55 - 225) = (345 - 300)/50 = 0.9\).

\(b_{xy} = (5 \cdot 69 - 15 \cdot 20)/(5 \cdot 90 - 400) = 45/50 = 0.9\).

Regression of Y on X: \(Y - 4 = 0.9(X - 3) \Rightarrow Y = 0.9 X + 1.3\).

Regression of X on Y: \(X - 3 = 0.9(Y - 4) \Rightarrow X = 0.9 Y - 0.6\).

\(r = \sqrt{0.9 \cdot 0.9} = 0.9\) (both coefficients positive ⇒ \(r > 0\)).

EXAMPLE 2 (Predicting from regression line)

For Example 1, predict \(Y\) when \(X = 6\):

\(\hat Y = 0.9(6) + 1.3 = 6.7\).

Predict \(X\) when \(Y = 7\): \(\hat X = 0.9(7) - 0.6 = 5.7\).

7. Correlation vs Regression

AspectCorrelationRegression
PurposeMeasures degree & direction of associationEstimates / predicts one variable from another
Symmetry\(r_{xy} = r_{yx}\) (symmetric)\(b_{yx} \ne b_{xy}\) in general (not symmetric)
Range\(-1 \le r \le 1\)Regression coefficients can be any real number
Cause & effectDoes not implyDoes not imply either; it assumes a direction (which variable explains which) that must come from outside the data
UnitsDimensionlessHas units (\(b_{yx}\) in units of Y/X)
Affected byBoth origin & scale changes?
Origin: No, Scale (same sign): No		Origin: No; Scale: Yes

8. Explained and Unexplained Variation

Define the total sum of squares of \(Y\):

\[ \text{SST} \;=\; \sum (y_i - \bar y)^2. \]

Decompose into:

\[ \text{SST} \;=\; \underbrace{\sum (\hat y_i - \bar y)^2}_{\text{Explained (SSR)}} \;+\; \underbrace{\sum (y_i - \hat y_i)^2}_{\text{Unexplained (SSE)}}. \]
\[ r^2 \;=\; \dfrac{\text{SSR}}{\text{SST}} \;=\; 1 - \dfrac{\text{SSE}}{\text{SST}}. \]

\(r^2\) is exactly the proportion of the total variation in \(Y\) explained by the regression on \(X\).

EXAMPLE 1

If SST = 200, SSR = 162, SSE = 38: \(r^2 = 162/200 = 0.81\); 81 % of variation is explained.

EXAMPLE 2

If \(r = 0.6\), then \(r^2 = 0.36\); only 36 % of variation in \(Y\) is explained — substantial unexplained variation remains.

Key Take-aways

Extra Practical Problems

PRACTICE

Additional worked problems with step-by-step procedures to support self-study, matching this unit's topics.

STEP-BY-STEP PROCEDURE
  1. Compute the regression coefficients \(b_{yx} = \dfrac{\sum(X-\bar X)(Y-\bar Y)}{\sum(X-\bar X)^2} = r\dfrac{\sigma_y}{\sigma_x}\) and \(b_{xy} = \dfrac{\sum(X-\bar X)(Y-\bar Y)}{\sum(Y-\bar Y)^2} = r\dfrac{\sigma_x}{\sigma_y}\).
  2. Form the regression lines: Y on X is \(Y - \bar Y = b_{yx}(X - \bar X)\); X on Y is \(X - \bar X = b_{xy}(Y - \bar Y)\).
  3. To estimate, substitute the known value into the appropriate line (use Y-on-X to predict Y, X-on-Y to predict X).
  4. Checks: \(b_{yx}\) and \(b_{xy}\) must have the same sign; \(r = \pm\sqrt{b_{yx}\,b_{xy}}\) (sign = sign of the coefficients); the two lines intersect at \((\bar X, \bar Y)\); one coefficient may exceed 1 but their product cannot.
  5. Test of significance of a regression coefficient uses a \(t\) statistic with \(n-2\) df.

Problem 1 — Are these valid regression lines?

DATA

Given lines \(Y = 5 + 2.8X\) and \(X = 3 - 0.5Y\).

Treating the first as the regression of \(Y\) on \(X\) gives \(b_{yx} = +2.8\); treating the second as the regression of \(X\) on \(Y\) gives \(b_{xy} = -0.5\). The two regression coefficients must have the same sign (both equal the sign of \(r\), since \(r = \pm\sqrt{b_{yx}\,b_{xy}}\) requires \(b_{yx}\,b_{xy} = r^2 \ge 0\)). Here \(b_{yx} = +2.8\) and \(b_{xy} = -0.5\) differ in sign — impossible. Hence these are not valid regression equations.

Problem 2 — Two Regression Lines: Means & Variance

DATA

Lines \(X + 2Y - 5 = 0\) and \(2X + 3Y - 8 = 0\); \(\sigma_x^2 = 12\).

Take line 1 as Y-on-X: \(Y = -0.5X + 2.5 \Rightarrow b_{yx} = -0.5\). Take line 2 as X-on-Y: \(X = -1.5Y + 4 \Rightarrow b_{xy} = -1.5\). Check: \(r = -\sqrt{b_{yx}\,b_{xy}} = -\sqrt{0.75} = -0.87\) (valid, in [−1, 1], signs consistent).

Means — both lines pass through \((\bar X, \bar Y)\); solving \(\bar X + 2\bar Y = 5\) and \(2\bar X + 3\bar Y = 8\) gives \(\bar X = 1,\ \bar Y = 2\).

Variance of Y — \(b_{yx} = r\dfrac{\sigma_y}{\sigma_x}\Rightarrow -0.5 = -0.87\cdot\dfrac{\sigma_y}{3.46}\) gives \(\sigma_y \approx 2\), so \(\sigma_y^2 = 4\).

Problem 3 — Regression Line & Estimation (means, SDs, r given)

DATA

Cinchona plants (n = 200): age X (mean 9.2, SD 2.1), yield Y (mean 16.5, SD 4.2), \(r = +0.84\).

\(b_{yx} = r\dfrac{\sigma_y}{\sigma_x} = 0.84\times\dfrac{4.2}{2.1} = 1.68\); line of Y on X: \(Y - 16.5 = 1.68(X - 9.2)\Rightarrow Y = 1.68X + 1.04\).

\(b_{xy} = r\dfrac{\sigma_x}{\sigma_y} = 0.84\times\dfrac{2.1}{4.2} = 0.42\); line of X on Y: \(X = 0.42Y + 2.27\).

Estimate yield for age 8: \(Y = 1.68(8) + 1.04 = \) 14.48 oz.

Problem 4 — Correlation, Both Regression Lines & Two-way Estimation

DATA

Stature (inches) of brother (X) and sister (Y) for 11 families (Pearson & Lee):

X7168666770717073726566
Y6964656365626564665962

\(\bar X = 69, \bar Y = 64\); \(\sum(X-\bar X)(Y-\bar Y) = 39\), \(\sum(X-\bar X)^2 = 74\), \(\sum(Y-\bar Y)^2 = 66\).

\(r = \dfrac{39}{\sqrt{74\times66}} = 0.558\). Test: \(t = \dfrac{0.558\sqrt{9}}{\sqrt{1-0.558^2}} = 2.018\) < \(t_{0.05,9} = 2.26\) → not significant.

\(b_{yx} = \dfrac{39}{74} = 0.527\) → \(Y - 64 = 0.527(X - 69)\); \(b_{xy} = \dfrac{39}{66} = 0.591\) → \(X - 69 = 0.591(Y - 64)\).

Estimate Y for X = 70: \(Y = 64 + 0.527(1) = 64.53\). Estimate X for Y = 62: \(X = 69 + 0.591(-2) = 67.82\).

Unsolved Exercises

PRACTICE
  1. Define the regression coefficient and state its properties.
  2. X (Economics) = 59,65,45,52,60,62,70,55,45,49; Y (Maths) = 75,70,55,65,60,69,80,65,59,61. Find both least-squares regression equations and estimate Y for X = 61. (Ans: \(Y-65.9 = 0.76(X-56.2)\); \(X-56.2 = 0.92(Y-65.9)\); Y = 69.54)
  3. Subject A: mean 39.5, SD 10.8; Subject B: mean 47.5, SD 16.8; \(r = 0.42\). Find both regression lines and estimate B for A = 50. (Ans: \(Y = 0.65X + 21.82\); \(X = 0.27Y + 26.67\); Y = 54.34)
  4. \(\bar X = 60, \bar Y = 141, \sum x^2 = 1000, \sum y^2 = 1936, \sum xy = 1380\) (deviation sums about the means). Find Y on X and estimate blood pressure at age 35. (Ans: \(b_{yx} = 1380/1000 = 1.38\), \(a = 141 - 1.38(60) = 58.2\), so \(Y = 1.38X + 58.2\); at \(X = 35\), Y = 106.5)
← Previous: Unit 3 Next: Unit 5 — Attributes →