Simple, Multiple and Logistic Regression
CSIR NET statistics: simple and multiple regression, R-squared, t and F tests, residual diagnostics, leverage, Cook's distance, VIF and logistic regression.
- 18 min
On this page · 13 sections
Prerequisites
- Gauss–Markov Models, BLUE and Linear Hypotheses: the hat matrix, \( s^2 = \text{SSE}/(n - r) \) and the general F test. Regression is the full-rank case.
- Descriptive Statistics and Exploratory Data Analysis: the two regression lines and \( r \) as descriptive tools.
- Inner Product Spaces and Orthonormal Bases: least squares as projection, the normal equations.
The Hook: A Good Fit Can Still Be Wrong
Simple Linear Regression
The \( x_i \) are treated as fixed. For tests and intervals, \( \varepsilon_i \overset{\text{iid}}{\sim} N(0, \sigma^2) \).
With \( S_{xx} = \sum(x_i - \bar x)^2 \), \( S_{xy} = \sum(x_i - \bar x)(y_i - \bar y) \), \( S_{yy} = \sum(y_i - \bar y)^2 \):
$$\boxed{\hat\beta_1 = \frac{S_{xy}}{S_{xx}}, \qquad \hat\beta_0 = \bar y - \hat\beta_1\bar x.}$$| Quantity | Formula |
|---|---|
| \( \operatorname{Var}(\hat\beta_1) \) | \( \frac{\sigma^2}{S_{xx}} \) |
| \( \operatorname{Var}(\hat\beta_0) \) | \( \sigma^2\left(\frac{1}{n} + \frac{\bar x^2}{S_{xx}}\right) \) |
| \( \operatorname{Cov}(\hat\beta_0, \hat\beta_1) \) | \( -\frac{\bar x\sigma^2}{S_{xx}} \) (zero iff \( \bar x = 0 \)) |
| SSR (regression SS) | \( \hat\beta_1S_{xy} = \frac{S_{xy}^2}{S_{xx}} \) |
| SSE | \( S_{yy} - \hat\beta_1S_{xy} \), df \( n - 2 \) |
| \( R^2 \) | \( \frac{\text{SSR}}{S_{yy}} = r^2 \) (square of the sample correlation) |
| slope test | \( t = \frac{\hat\beta_1}{s/\sqrt{S_{xx}}} \sim t_{n - 2} \); \( t^2 = F = \frac{\text{SSR}}{s^2} \) |
| mean response at \( x_0 \) | \( \hat y_0 \pm t_{n-2}\, s\sqrt{\frac{1}{n} + \frac{(x_0 - \bar x)^2}{S_{xx}}} \) |
| new observation at \( x_0 \) | \( \hat y_0 \pm t_{n-2}\, s\sqrt{1 + \frac{1}{n} + \frac{(x_0 - \bar x)^2}{S_{xx}}} \) |
Properties of the fitted line (with an intercept): it passes through \( (\bar x, \bar y) \); \( \sum e_i = 0 \); \( \sum x_ie_i = 0 \); \( \sum\hat y_ie_i = 0 \). Both intervals are narrowest at \( x_0 = \bar x \) and widen as \( x_0 \) moves away. Regression through the origin (\( y = \beta x + \varepsilon \)): \( \hat\beta = \frac{\sum x_iy_i}{\sum x_i^2} \), residuals need not sum to zero.
Multiple Linear Regression
\( Y = X\beta + \varepsilon \), where \( X \) is \( n \times p \) of full column rank \( p \) (first column all ones for the intercept, then \( p - 1 \) predictors).
$$\hat\beta = (X^TX)^{-1}X^TY, \qquad \operatorname{Var}(\hat\beta) = \sigma^2(X^TX)^{-1}, \qquad H = X(X^TX)^{-1}X^T, \qquad s^2 = \frac{\text{SSE}}{n - p}.$$Sums of squares and R²
\( \text{SST} = \sum(y_i - \bar y)^2 = \text{SSR} + \text{SSE} \), with df \( n - 1 = (p - 1) + (n - p) \) when the model has an intercept.
| Measure | Formula | Behaviour |
|---|---|---|
| \( R^2 \) | \( \frac{\text{SSR}}{\text{SST}} = 1 - \frac{\text{SSE}}{\text{SST}} \) | never decreases when a predictor is added |
| adjusted \( R^2 \) | \( 1 - \frac{\text{SSE}/(n - p)}{\text{SST}/(n - 1)} = 1 - (1 - R^2)\frac{n - 1}{n - p} \) | can fall, and can be negative |
| multiple correlation | \( R = \operatorname{corr}(y, \hat y) \) | equals the sample multiple correlation of \( y \) on the predictors |
| overall F | \( \frac{\text{SSR}/(p - 1)}{\text{SSE}/(n - p)} = \frac{R^2/(p - 1)}{(1 - R^2)/(n - p)} \sim F_{p - 1, n - p} \) | tests all slopes zero |
| partial F (drop \( q \) predictors) | \( \frac{(\text{SSE}_{\text{reduced}} - \text{SSE}_{\text{full}})/q}{\text{SSE}_{\text{full}}/(n - p)} \) | the general linear hypothesis test |
| individual t | \( \frac{\hat\beta_j}{s\sqrt{c_{jj}}} \sim t_{n - p} \), \( c_{jj} = [(X^TX)^{-1}]_{jj} \) | \( t^2 \) = partial F for that one variable |
Here \( p \) counts all parameters including the intercept. Some books write \( k \) predictors and \( n - k - 1 \) error df; it is the same thing.
Regression Diagnostics
The fitted model is only as good as its assumptions: linearity, constant variance, independence, normality, and no single point dominating the fit.
Residuals and leverage
\( e = (I - H)Y \), so \( E(e) = 0 \) and \( \operatorname{Var}(e) = \sigma^2(I - H) \). Residuals are correlated and have unequal variances \( \sigma^2(1 - h_{ii}) \), even when the errors are iid.
| Quantity | Formula | Rule of thumb |
|---|---|---|
| leverage \( h_{ii} \) | \( i \)-th diagonal of \( H \); \( \sum h_{ii} = p \), mean \( \frac{p}{n} \) | high if \( h_{ii} > \frac{2p}{n} \) |
| bounds | \( \frac{1}{n} \le h_{ii} \le 1 \) (model with intercept) | |
| simple regression | \( h_{ii} = \frac{1}{n} + \frac{(x_i - \bar x)^2}{S_{xx}} \) | far-out \( x \) means high leverage |
| standardised (internally studentised) residual | \( r_i = \frac{e_i}{s\sqrt{1 - h_{ii}}} \) | \( \lvert r_i \rvert > 2 \) or 3 is suspicious |
| externally studentised residual | \( t_i = \frac{e_i}{s_{(i)}\sqrt{1 - h_{ii}}} \), \( s_{(i)} \) from the fit without point \( i \) | \( \sim t_{n - p - 1} \) under the model |
| Cook’s distance | \( D_i = \frac{r_i^2}{p}\cdot\frac{h_{ii}}{1 - h_{ii}} \) | large if \( D_i \) near or above 1 |
Leverage depends only on \( X \): it measures potential influence. Influence (Cook’s distance) needs both leverage and a sizable residual. A high-leverage point that lies on the line has a small residual and little influence.
In the worked example, \( h_{ii} = 0.6, 0.3, 0.2, 0.3, 0.6 \) (sum \( 2 = p \)): the end points \( x = 1 \) and \( x = 5 \) have the most leverage, and the middle point the least.
Assumption checks
| Problem | What you see | Remedy |
|---|---|---|
| non-linearity | curved pattern in residuals vs fitted or vs \( x_j \) | add terms (\( x^2 \)), transform |
| heteroscedasticity | funnel shape in residuals vs fitted | transform \( y \) (log, square root, Box–Cox), weighted least squares |
| non-normality | curved normal Q–Q plot of residuals | transform; large-sample robustness for \( \hat\beta \) |
| autocorrelation (time order) | runs in residuals vs time; Durbin–Watson | model the correlation (GLS) |
| multicollinearity | large standard errors, unstable signs; high VIF | drop or combine predictors, ridge, principal components |
| outliers, influential points | large \( \lvert t_i \rvert \), large \( D_i \) | check the data; robust methods |
Durbin–Watson statistic for first-order autocorrelation of residuals in time order:
$$d = \frac{\sum_{t=2}^n(e_t - e_{t-1})^2}{\sum_{t=1}^n e_t^2} \approx 2(1 - \hat\rho), \qquad 0 \le d \le 4.$$\( d \approx 2 \): no autocorrelation; \( d \) near 0: strong positive; \( d \) near 4: strong negative.
Variance inflation factor. Regress \( x_j \) on the other predictors and call the coefficient of determination \( R_j^2 \). Then
$$\operatorname{Var}(\hat\beta_j) = \frac{\sigma^2}{S_{x_jx_j}}\cdot\underbrace{\frac{1}{1 - R_j^2}}_{\text{VIF}_j}.$$\( \text{VIF}_j = 1 \) for a predictor uncorrelated with the others; a common warning level is 10 (\( R_j^2 = 0.9 \)). For two predictors with correlation \( r_{12} \), both VIFs equal \( \frac{1}{1 - r_{12}^2} \).
Logistic Regression
For a binary response \( Y \in \{0, 1\} \) with \( p(x) = P(Y = 1 \mid x) \), linear regression fails: \( x^T\beta \) is unbounded, and \( \operatorname{Var}(Y) = p(1 - p) \) is not constant. Logistic regression models the log-odds:
The \( Y_i \) are independent Bernoulli\( (p(x_i)) \). It is a generalised linear model with binomial response and logit link.
Interpretation. A unit increase in \( x_j \), others fixed, adds \( \beta_j \) to the log-odds, so it multiplies the odds by \( e^{\beta_j} \): the odds ratio. It does not multiply the probability by \( e^{\beta_j} \), and the change in probability depends on where you start. \( p = \frac{1}{2} \) exactly when \( x^T\beta = 0 \). For a single binary predictor, \( \hat\beta_1 \) is the log of the sample odds ratio of the \( 2 \times 2 \) table.
Estimation by maximum likelihood. The log-likelihood is
$$\ell(\beta) = \sum_i\left[y_i\,x_i^T\beta - \ln\left(1 + e^{x_i^T\beta}\right)\right], \qquad \text{score: } X^T(y - \hat p) = 0.$$There is no closed form; the equations are solved by Newton–Raphson, which here is iteratively reweighted least squares, with weights \( W = \operatorname{diag}(\hat p_i(1 - \hat p_i)) \). The large-sample covariance of \( \hat\beta \) is \( (X^TWX)^{-1} \). If the two classes are completely separated by a hyperplane in \( x \), the MLE does not exist (the likelihood increases without bound as \( \lVert\beta\rVert \to \infty \)).
Inference. Wald test \( z = \frac{\hat\beta_j}{\text{SE}(\hat\beta_j)} \); likelihood ratio test \( G^2 = -2(\ell_{\text{reduced}} - \ell_{\text{full}}) \sim \chi^2_q \) approximately; deviance \( D = -2(\ell_{\text{model}} - \ell_{\text{saturated}}) \) replaces SSE.
graph TD
A["Response type?"] -->|continuous| B["Linear regression
OLS: b = (X'X)^-1 X'y"]
A -->|binary 0/1| C["Logistic regression
logit p = x'beta, MLE by IRLS"]
B --> D["Check: residual plots, Q-Q,
leverage h_ii, Cook's D, VIF, DW"]
B --> E["Test: t for one coefficient,
F overall or partial"]
C --> F["Interpret e^beta as odds ratio;
Wald or likelihood ratio tests"]Memory Tricks
- “S-x-y over S-x-x.” Slope \( = \frac{S_{xy}}{S_{xx}} \); the line goes through the means.
- Leverages add to p. \( \operatorname{tr}H = p \); average \( \frac{p}{n} \); flag twice the average.
- Adjusted R² charges rent. Each predictor costs a degree of freedom: \( 1 - (1 - R^2)\frac{n - 1}{n - p} \).
- DW = 2 means calm. \( d \approx 2(1 - \hat\rho) \): 0 for \( \hat\rho = 1 \), 4 for \( \hat\rho = -1 \).
- Logistic betas are log odds ratios. Exponentiate to talk about odds, never about probabilities directly.
Don’t Confuse
| This | Is not this | The difference |
|---|---|---|
| leverage \( h_{ii} \) | influence (Cook’s \( D_i \)) | leverage is about \( x \) only; influence needs a residual too |
| confidence interval for \( E(y_0) \) | prediction interval for \( y_0 \) | the prediction interval has the extra “\( 1 + \)” under the root |
| \( R^2 \) | adjusted \( R^2 \) | \( R^2 \) never falls when you add a variable |
| errors \( \varepsilon_i \) (iid) | residuals \( e_i \) | residuals are correlated, with variance \( \sigma^2(1 - h_{ii}) \) |
| odds ratio \( e^{\beta_j} \) | relative risk | they are close only when \( p \) is small |
| error df \( n - p \) (\( p \) parameters) | \( n - k - 1 \) (\( k \) predictors) | same number, different bookkeeping |
| high VIF | a wrong model | collinearity inflates variances; the fitted values can still be fine |
Common Mistakes
Practice
All questions below are original practice questions written for this page. They are not from any CSIR NET paper. In Level 3, as in Part C, one or more options may be correct.
Level 1: Foundation
- Write the least squares slope in terms of \( S_{xy} \) and \( S_{xx} \).
- In a regression with \( p \) parameters (including the intercept), what is \( \sum_i h_{ii} \)?
- Define the logit of a probability \( p \).
Answers
- \( \hat\beta_1 = \frac{S_{xy}}{S_{xx}} \).
- \( p \), the trace of the hat matrix.
- \( \ln\frac{p}{1 - p} \), the log-odds.
Level 2: Part B Style
Practice Q1. For \( x = 1, 2, 3, 4, 5 \) and \( y = 2, 3, 5, 4, 6 \), the least squares line of \( y \) on \( x \) is:
(a) \( y = 1.3 + 0.9x \) (b) \( y = 1 + x \) (c) \( y = 0.9 + 1.3x \) (d) \( y = 1.5 + 0.8x \)
Answer: (a)
\( S_{xy} = (-2)(-2) + (-1)(-1) + 0 + (1)(0) + (2)(2) = 9 \), \( S_{xx} = 10 \): \( \hat\beta_1 = 0.9 \), \( \hat\beta_0 = 4 - 0.9 \times 3 = 1.3 \).
Practice Q2. A regression of \( y \) on 4 predictors with an intercept, fitted to \( n = 25 \) observations, has \( R^2 = 0.8 \). The adjusted \( R^2 \) is:
(a) 0.75 (b) 0.76 (c) 0.78 (d) 0.79
Answer: (b)
\( p = 5 \). \( 1 - (1 - 0.8)\frac{24}{20} = 1 - 0.24 = 0.76 \). (The overall F is \( \frac{0.8/4}{0.2/20} = 20 \) on \( (4, 20) \) df.)
Practice Q3. When predictor \( x_3 \) is regressed on the other predictors, the coefficient of determination is 0.75. The variance inflation factor of \( x_3 \) is:
(a) 1.33 (b) 0.25 (c) 4 (d) 7.5
Answer: (c)
\( \text{VIF} = \frac{1}{1 - 0.75} = 4 \): the variance of \( \hat\beta_3 \) is four times what it would be with \( x_3 \) uncorrelated with the others.
Practice Q4. A fitted logistic model is \( \operatorname{logit} p = -3 + 0.5x \). The fitted probability at \( x = 4 \) is closest to:
(a) 0.12 (b) 0.27 (c) 0.37 (d) 0.73
Answer: (b)
Log-odds \( -3 + 2 = -1 \); \( p = \frac{1}{1 + e^{1}} = \frac{1}{3.718} \approx 0.269 \). Option (d) is \( 1 - p \); (c) is \( e^{-1} \), the odds.
Level 3: Part C Style
Practice Q5. For the data \( x = 1, 2, 3, 4, 5 \), \( y = 2, 3, 5, 4, 6 \) fitted by simple linear regression with intercept, which statements are true?
(a) \( R^2 = 0.81 \). (b) The leverage of the point with \( x = 5 \) is 0.6. (c) The unbiased estimate of \( \sigma^2 \) is 0.38. (d) The F statistic for \( H_0: \beta_1 = 0 \) equals the square of the t statistic for the same hypothesis.
Answer: (a), (b) and (d)
- (a) True. \( \text{SSR} = \frac{81}{10} = 8.1 \), \( S_{yy} = 10 \).
- (b) True. \( h = \frac{1}{5} + \frac{(5 - 3)^2}{10} = 0.6 \).
- (c) False. \( s^2 = \frac{\text{SSE}}{n - 2} = \frac{1.9}{3} \approx 0.633 \). The value 0.38 divides by \( n = 5 \).
- (d) True. With one numerator degree of freedom, \( F = t^2 \) (here about 12.79).
Practice Q6. In a multiple linear regression \( Y = X\beta + \varepsilon \) with intercept, \( X \) of order \( n \times p \) and full column rank, and \( \varepsilon \sim N_n(0, \sigma^2I) \), let \( H \) be the hat matrix and \( e = (I - H)Y \). Which statements are true?
(a) \( \operatorname{Var}(e_i) = \sigma^2(1 - h_{ii}) \). (b) The residuals \( e_1, \dots, e_n \) are mutually independent. (c) \( \sum_i e_i = 0 \). (d) A Durbin–Watson statistic close to 2 indicates strong positive autocorrelation.
Answer: (a) and (c)
- (a) True. \( \operatorname{Var}(e) = \sigma^2(I - H) \).
- (b) False. \( \operatorname{Cov}(e_i, e_j) = -\sigma^2h_{ij} \), generally non-zero; and \( e \) has a singular covariance of rank \( n - p \).
- (c) True. \( X^Te = 0 \) and the first column of \( X \) is all ones.
- (d) False. \( d \approx 2(1 - \hat\rho) \): \( d \approx 2 \) means \( \hat\rho \approx 0 \). Strong positive autocorrelation pushes \( d \) towards 0.
Practice Q7. In a study, 30 of 100 exposed and 10 of 100 unexposed subjects develop a condition. A logistic regression of the condition on an exposure indicator \( x \) (1 = exposed) is fitted by maximum likelihood. Which statements are true?
(a) \( \hat\beta_1 = \ln\frac{27}{7} \approx 1.35 \). (b) \( e^{\hat\beta_1} \) is the estimated ratio of the probabilities of the condition, exposed to unexposed. (c) The fitted probability for an exposed subject is 0.3. (d) \( \hat\beta_0 = \ln\frac{1}{9} \).
Answer: (a), (c) and (d)
With one binary predictor the model is saturated (two parameters, two groups), so fitted probabilities equal observed proportions.
- (a) True. Odds ratio \( \frac{30/70}{10/90} = \frac{27}{7} \approx 3.857 \); its log is about 1.350.
- (b) False. \( e^{\hat\beta_1} \) is the odds ratio, 3.857. The ratio of probabilities (relative risk) is \( \frac{0.3}{0.1} = 3 \).
- (c) True. Fitted \( p = 0.3 \) for \( x = 1 \).
- (d) True. \( \hat\beta_0 \) is the log-odds for \( x = 0 \): \( \ln\frac{0.1}{0.9} = \ln\frac{1}{9} \approx -2.197 \).
Quick Revision Table
| Idea | Fact |
|---|---|
| Slope, intercept | \( \frac{S_{xy}}{S_{xx}} \), \( \bar y - \hat\beta_1\bar x \) |
| Slope variance | \( \frac{\sigma^2}{S_{xx}} \); error df \( n - 2 \) |
| \( R^2 \) (simple) | \( r^2 = \frac{S_{xy}^2}{S_{xx}S_{yy}} \) |
| Multiple | \( \hat\beta = (X^TX)^{-1}X^TY \), \( \operatorname{Var} = \sigma^2(X^TX)^{-1} \), df \( n - p \) |
| Adjusted \( R^2 \) | \( 1 - (1 - R^2)\frac{n - 1}{n - p} \) |
| Overall F | \( \frac{R^2/(p - 1)}{(1 - R^2)/(n - p)} \) |
| Leverage | \( h_{ii} \in [\frac{1}{n}, 1] \), \( \sum h_{ii} = p \), flag \( > \frac{2p}{n} \) |
| Residual variance | \( \sigma^2(1 - h_{ii}) \) |
| Cook’s distance | \( \frac{r_i^2}{p}\cdot\frac{h_{ii}}{1 - h_{ii}} \) |
| VIF | \( \frac{1}{1 - R_j^2} \) |
| Durbin–Watson | \( \approx 2(1 - \hat\rho) \), range \( [0, 4] \) |
| Logistic | \( \ln\frac{p}{1 - p} = x^T\beta \); \( e^{\beta_j} \) = odds ratio; MLE by IRLS |
Key Takeaways
What’s Next
- Partial and Multiple Correlation: \( R^2 \) as a squared multiple correlation and the partial correlation behind each coefficient.
- Multivariate Normal, Wishart and Quadratic Forms: conditional normal distributions are linear regressions.
- ANOVA, ANCOVA and Fixed, Random and Mixed Effects: regression with indicator variables.
- Back to the Linear Models and Multivariate Analysis chapter overview.