Skip to content

Simple, Multiple and Logistic Regression

CSIR NET statistics: simple and multiple regression, R-squared, t and F tests, residual diagnostics, leverage, Cook's distance, VIF and logistic regression.

  • 18 min read
  • Updated
On this page · 13 sections
  1. Prerequisites
  2. The Hook: A Good Fit Can Still Be Wrong
  3. Simple Linear Regression
  4. Multiple Linear Regression
  5. Regression Diagnostics
  6. Logistic Regression
  7. Memory Tricks
  8. Don’t Confuse
  9. Common Mistakes
  10. Practice
  11. Quick Revision Table
  12. Key Takeaways
  13. What’s Next

Prerequisites

The Hook: A Good Fit Can Still Be Wrong

Simple Linear Regression

Simple linear regression model
$$y_i = \beta_0 + \beta_1x_i + \varepsilon_i, \qquad i = 1, \dots, n, \qquad E\varepsilon_i = 0, \; \operatorname{Var}\varepsilon_i = \sigma^2, \; \text{uncorrelated}.$$

The \( x_i \) are treated as fixed. For tests and intervals, \( \varepsilon_i \overset{\text{iid}}{\sim} N(0, \sigma^2) \).

With \( S_{xx} = \sum(x_i - \bar x)^2 \), \( S_{xy} = \sum(x_i - \bar x)(y_i - \bar y) \), \( S_{yy} = \sum(y_i - \bar y)^2 \):

$$\boxed{\hat\beta_1 = \frac{S_{xy}}{S_{xx}}, \qquad \hat\beta_0 = \bar y - \hat\beta_1\bar x.}$$
QuantityFormula
\( \operatorname{Var}(\hat\beta_1) \)\( \frac{\sigma^2}{S_{xx}} \)
\( \operatorname{Var}(\hat\beta_0) \)\( \sigma^2\left(\frac{1}{n} + \frac{\bar x^2}{S_{xx}}\right) \)
\( \operatorname{Cov}(\hat\beta_0, \hat\beta_1) \)\( -\frac{\bar x\sigma^2}{S_{xx}} \) (zero iff \( \bar x = 0 \))
SSR (regression SS)\( \hat\beta_1S_{xy} = \frac{S_{xy}^2}{S_{xx}} \)
SSE\( S_{yy} - \hat\beta_1S_{xy} \), df \( n - 2 \)
\( R^2 \)\( \frac{\text{SSR}}{S_{yy}} = r^2 \) (square of the sample correlation)
slope test\( t = \frac{\hat\beta_1}{s/\sqrt{S_{xx}}} \sim t_{n - 2} \); \( t^2 = F = \frac{\text{SSR}}{s^2} \)
mean response at \( x_0 \)\( \hat y_0 \pm t_{n-2}\, s\sqrt{\frac{1}{n} + \frac{(x_0 - \bar x)^2}{S_{xx}}} \)
new observation at \( x_0 \)\( \hat y_0 \pm t_{n-2}\, s\sqrt{1 + \frac{1}{n} + \frac{(x_0 - \bar x)^2}{S_{xx}}} \)

Properties of the fitted line (with an intercept): it passes through \( (\bar x, \bar y) \); \( \sum e_i = 0 \); \( \sum x_ie_i = 0 \); \( \sum\hat y_ie_i = 0 \). Both intervals are narrowest at \( x_0 = \bar x \) and widen as \( x_0 \) moves away. Regression through the origin (\( y = \beta x + \varepsilon \)): \( \hat\beta = \frac{\sum x_iy_i}{\sum x_i^2} \), residuals need not sum to zero.

Multiple Linear Regression

\( Y = X\beta + \varepsilon \), where \( X \) is \( n \times p \) of full column rank \( p \) (first column all ones for the intercept, then \( p - 1 \) predictors).

$$\hat\beta = (X^TX)^{-1}X^TY, \qquad \operatorname{Var}(\hat\beta) = \sigma^2(X^TX)^{-1}, \qquad H = X(X^TX)^{-1}X^T, \qquad s^2 = \frac{\text{SSE}}{n - p}.$$

Sums of squares and R²

\( \text{SST} = \sum(y_i - \bar y)^2 = \text{SSR} + \text{SSE} \), with df \( n - 1 = (p - 1) + (n - p) \) when the model has an intercept.

MeasureFormulaBehaviour
\( R^2 \)\( \frac{\text{SSR}}{\text{SST}} = 1 - \frac{\text{SSE}}{\text{SST}} \)never decreases when a predictor is added
adjusted \( R^2 \)\( 1 - \frac{\text{SSE}/(n - p)}{\text{SST}/(n - 1)} = 1 - (1 - R^2)\frac{n - 1}{n - p} \)can fall, and can be negative
multiple correlation\( R = \operatorname{corr}(y, \hat y) \)equals the sample multiple correlation of \( y \) on the predictors
overall F\( \frac{\text{SSR}/(p - 1)}{\text{SSE}/(n - p)} = \frac{R^2/(p - 1)}{(1 - R^2)/(n - p)} \sim F_{p - 1, n - p} \)tests all slopes zero
partial F (drop \( q \) predictors)\( \frac{(\text{SSE}_{\text{reduced}} - \text{SSE}_{\text{full}})/q}{\text{SSE}_{\text{full}}/(n - p)} \)the general linear hypothesis test
individual t\( \frac{\hat\beta_j}{s\sqrt{c_{jj}}} \sim t_{n - p} \), \( c_{jj} = [(X^TX)^{-1}]_{jj} \)\( t^2 \) = partial F for that one variable

Here \( p \) counts all parameters including the intercept. Some books write \( k \) predictors and \( n - k - 1 \) error df; it is the same thing.

Regression Diagnostics

The fitted model is only as good as its assumptions: linearity, constant variance, independence, normality, and no single point dominating the fit.

Residuals and leverage

\( e = (I - H)Y \), so \( E(e) = 0 \) and \( \operatorname{Var}(e) = \sigma^2(I - H) \). Residuals are correlated and have unequal variances \( \sigma^2(1 - h_{ii}) \), even when the errors are iid.

QuantityFormulaRule of thumb
leverage \( h_{ii} \)\( i \)-th diagonal of \( H \); \( \sum h_{ii} = p \), mean \( \frac{p}{n} \)high if \( h_{ii} > \frac{2p}{n} \)
bounds\( \frac{1}{n} \le h_{ii} \le 1 \) (model with intercept)
simple regression\( h_{ii} = \frac{1}{n} + \frac{(x_i - \bar x)^2}{S_{xx}} \)far-out \( x \) means high leverage
standardised (internally studentised) residual\( r_i = \frac{e_i}{s\sqrt{1 - h_{ii}}} \)\( \lvert r_i \rvert > 2 \) or 3 is suspicious
externally studentised residual\( t_i = \frac{e_i}{s_{(i)}\sqrt{1 - h_{ii}}} \), \( s_{(i)} \) from the fit without point \( i \)\( \sim t_{n - p - 1} \) under the model
Cook’s distance\( D_i = \frac{r_i^2}{p}\cdot\frac{h_{ii}}{1 - h_{ii}} \)large if \( D_i \) near or above 1

Leverage depends only on \( X \): it measures potential influence. Influence (Cook’s distance) needs both leverage and a sizable residual. A high-leverage point that lies on the line has a small residual and little influence.

In the worked example, \( h_{ii} = 0.6, 0.3, 0.2, 0.3, 0.6 \) (sum \( 2 = p \)): the end points \( x = 1 \) and \( x = 5 \) have the most leverage, and the middle point the least.

Assumption checks

ProblemWhat you seeRemedy
non-linearitycurved pattern in residuals vs fitted or vs \( x_j \)add terms (\( x^2 \)), transform
heteroscedasticityfunnel shape in residuals vs fittedtransform \( y \) (log, square root, Box–Cox), weighted least squares
non-normalitycurved normal Q–Q plot of residualstransform; large-sample robustness for \( \hat\beta \)
autocorrelation (time order)runs in residuals vs time; Durbin–Watsonmodel the correlation (GLS)
multicollinearitylarge standard errors, unstable signs; high VIFdrop or combine predictors, ridge, principal components
outliers, influential pointslarge \( \lvert t_i \rvert \), large \( D_i \)check the data; robust methods

Durbin–Watson statistic for first-order autocorrelation of residuals in time order:

$$d = \frac{\sum_{t=2}^n(e_t - e_{t-1})^2}{\sum_{t=1}^n e_t^2} \approx 2(1 - \hat\rho), \qquad 0 \le d \le 4.$$

\( d \approx 2 \): no autocorrelation; \( d \) near 0: strong positive; \( d \) near 4: strong negative.

Variance inflation factor. Regress \( x_j \) on the other predictors and call the coefficient of determination \( R_j^2 \). Then

$$\operatorname{Var}(\hat\beta_j) = \frac{\sigma^2}{S_{x_jx_j}}\cdot\underbrace{\frac{1}{1 - R_j^2}}_{\text{VIF}_j}.$$

\( \text{VIF}_j = 1 \) for a predictor uncorrelated with the others; a common warning level is 10 (\( R_j^2 = 0.9 \)). For two predictors with correlation \( r_{12} \), both VIFs equal \( \frac{1}{1 - r_{12}^2} \).

Logistic Regression

For a binary response \( Y \in \{0, 1\} \) with \( p(x) = P(Y = 1 \mid x) \), linear regression fails: \( x^T\beta \) is unbounded, and \( \operatorname{Var}(Y) = p(1 - p) \) is not constant. Logistic regression models the log-odds:

Logistic regression model
$$\operatorname{logit} p(x) = \ln\frac{p(x)}{1 - p(x)} = \beta_0 + \beta_1x_1 + \dots + \beta_kx_k, \qquad p(x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1x_1 + \dots + \beta_kx_k)}}.$$

The \( Y_i \) are independent Bernoulli\( (p(x_i)) \). It is a generalised linear model with binomial response and logit link.

Interpretation. A unit increase in \( x_j \), others fixed, adds \( \beta_j \) to the log-odds, so it multiplies the odds by \( e^{\beta_j} \): the odds ratio. It does not multiply the probability by \( e^{\beta_j} \), and the change in probability depends on where you start. \( p = \frac{1}{2} \) exactly when \( x^T\beta = 0 \). For a single binary predictor, \( \hat\beta_1 \) is the log of the sample odds ratio of the \( 2 \times 2 \) table.

Estimation by maximum likelihood. The log-likelihood is

$$\ell(\beta) = \sum_i\left[y_i\,x_i^T\beta - \ln\left(1 + e^{x_i^T\beta}\right)\right], \qquad \text{score: } X^T(y - \hat p) = 0.$$

There is no closed form; the equations are solved by Newton–Raphson, which here is iteratively reweighted least squares, with weights \( W = \operatorname{diag}(\hat p_i(1 - \hat p_i)) \). The large-sample covariance of \( \hat\beta \) is \( (X^TWX)^{-1} \). If the two classes are completely separated by a hyperplane in \( x \), the MLE does not exist (the likelihood increases without bound as \( \lVert\beta\rVert \to \infty \)).

Inference. Wald test \( z = \frac{\hat\beta_j}{\text{SE}(\hat\beta_j)} \); likelihood ratio test \( G^2 = -2(\ell_{\text{reduced}} - \ell_{\text{full}}) \sim \chi^2_q \) approximately; deviance \( D = -2(\ell_{\text{model}} - \ell_{\text{saturated}}) \) replaces SSE.

graph TD
    A["Response type?"] -->|continuous| B["Linear regression
OLS: b = (X'X)^-1 X'y"] A -->|binary 0/1| C["Logistic regression
logit p = x'beta, MLE by IRLS"] B --> D["Check: residual plots, Q-Q,
leverage h_ii, Cook's D, VIF, DW"] B --> E["Test: t for one coefficient,
F overall or partial"] C --> F["Interpret e^beta as odds ratio;
Wald or likelihood ratio tests"]

Memory Tricks

  • “S-x-y over S-x-x.” Slope \( = \frac{S_{xy}}{S_{xx}} \); the line goes through the means.
  • Leverages add to p. \( \operatorname{tr}H = p \); average \( \frac{p}{n} \); flag twice the average.
  • Adjusted R² charges rent. Each predictor costs a degree of freedom: \( 1 - (1 - R^2)\frac{n - 1}{n - p} \).
  • DW = 2 means calm. \( d \approx 2(1 - \hat\rho) \): 0 for \( \hat\rho = 1 \), 4 for \( \hat\rho = -1 \).
  • Logistic betas are log odds ratios. Exponentiate to talk about odds, never about probabilities directly.

Don’t Confuse

ThisIs not thisThe difference
leverage \( h_{ii} \)influence (Cook’s \( D_i \))leverage is about \( x \) only; influence needs a residual too
confidence interval for \( E(y_0) \)prediction interval for \( y_0 \)the prediction interval has the extra “\( 1 + \)” under the root
\( R^2 \)adjusted \( R^2 \)\( R^2 \) never falls when you add a variable
errors \( \varepsilon_i \) (iid)residuals \( e_i \)residuals are correlated, with variance \( \sigma^2(1 - h_{ii}) \)
odds ratio \( e^{\beta_j} \)relative riskthey are close only when \( p \) is small
error df \( n - p \) (\( p \) parameters)\( n - k - 1 \) (\( k \) predictors)same number, different bookkeeping
high VIFa wrong modelcollinearity inflates variances; the fitted values can still be fine

Common Mistakes

Practice

All questions below are original practice questions written for this page. They are not from any CSIR NET paper. In Level 3, as in Part C, one or more options may be correct.

Level 1: Foundation

  1. Write the least squares slope in terms of \( S_{xy} \) and \( S_{xx} \).
  2. In a regression with \( p \) parameters (including the intercept), what is \( \sum_i h_{ii} \)?
  3. Define the logit of a probability \( p \).
Answers
  1. \( \hat\beta_1 = \frac{S_{xy}}{S_{xx}} \).
  2. \( p \), the trace of the hat matrix.
  3. \( \ln\frac{p}{1 - p} \), the log-odds.

Level 2: Part B Style

Practice Q1. For \( x = 1, 2, 3, 4, 5 \) and \( y = 2, 3, 5, 4, 6 \), the least squares line of \( y \) on \( x \) is:

(a) \( y = 1.3 + 0.9x \) (b) \( y = 1 + x \) (c) \( y = 0.9 + 1.3x \) (d) \( y = 1.5 + 0.8x \)

Answer: (a)

\( S_{xy} = (-2)(-2) + (-1)(-1) + 0 + (1)(0) + (2)(2) = 9 \), \( S_{xx} = 10 \): \( \hat\beta_1 = 0.9 \), \( \hat\beta_0 = 4 - 0.9 \times 3 = 1.3 \).

Practice Q2. A regression of \( y \) on 4 predictors with an intercept, fitted to \( n = 25 \) observations, has \( R^2 = 0.8 \). The adjusted \( R^2 \) is:

(a) 0.75 (b) 0.76 (c) 0.78 (d) 0.79

Answer: (b)

\( p = 5 \). \( 1 - (1 - 0.8)\frac{24}{20} = 1 - 0.24 = 0.76 \). (The overall F is \( \frac{0.8/4}{0.2/20} = 20 \) on \( (4, 20) \) df.)

Practice Q3. When predictor \( x_3 \) is regressed on the other predictors, the coefficient of determination is 0.75. The variance inflation factor of \( x_3 \) is:

(a) 1.33 (b) 0.25 (c) 4 (d) 7.5

Answer: (c)

\( \text{VIF} = \frac{1}{1 - 0.75} = 4 \): the variance of \( \hat\beta_3 \) is four times what it would be with \( x_3 \) uncorrelated with the others.

Practice Q4. A fitted logistic model is \( \operatorname{logit} p = -3 + 0.5x \). The fitted probability at \( x = 4 \) is closest to:

(a) 0.12 (b) 0.27 (c) 0.37 (d) 0.73

Answer: (b)

Log-odds \( -3 + 2 = -1 \); \( p = \frac{1}{1 + e^{1}} = \frac{1}{3.718} \approx 0.269 \). Option (d) is \( 1 - p \); (c) is \( e^{-1} \), the odds.

Level 3: Part C Style

Practice Q5. For the data \( x = 1, 2, 3, 4, 5 \), \( y = 2, 3, 5, 4, 6 \) fitted by simple linear regression with intercept, which statements are true?

(a) \( R^2 = 0.81 \). (b) The leverage of the point with \( x = 5 \) is 0.6. (c) The unbiased estimate of \( \sigma^2 \) is 0.38. (d) The F statistic for \( H_0: \beta_1 = 0 \) equals the square of the t statistic for the same hypothesis.

Answer: (a), (b) and (d)
  • (a) True. \( \text{SSR} = \frac{81}{10} = 8.1 \), \( S_{yy} = 10 \).
  • (b) True. \( h = \frac{1}{5} + \frac{(5 - 3)^2}{10} = 0.6 \).
  • (c) False. \( s^2 = \frac{\text{SSE}}{n - 2} = \frac{1.9}{3} \approx 0.633 \). The value 0.38 divides by \( n = 5 \).
  • (d) True. With one numerator degree of freedom, \( F = t^2 \) (here about 12.79).

Practice Q6. In a multiple linear regression \( Y = X\beta + \varepsilon \) with intercept, \( X \) of order \( n \times p \) and full column rank, and \( \varepsilon \sim N_n(0, \sigma^2I) \), let \( H \) be the hat matrix and \( e = (I - H)Y \). Which statements are true?

(a) \( \operatorname{Var}(e_i) = \sigma^2(1 - h_{ii}) \). (b) The residuals \( e_1, \dots, e_n \) are mutually independent. (c) \( \sum_i e_i = 0 \). (d) A Durbin–Watson statistic close to 2 indicates strong positive autocorrelation.

Answer: (a) and (c)
  • (a) True. \( \operatorname{Var}(e) = \sigma^2(I - H) \).
  • (b) False. \( \operatorname{Cov}(e_i, e_j) = -\sigma^2h_{ij} \), generally non-zero; and \( e \) has a singular covariance of rank \( n - p \).
  • (c) True. \( X^Te = 0 \) and the first column of \( X \) is all ones.
  • (d) False. \( d \approx 2(1 - \hat\rho) \): \( d \approx 2 \) means \( \hat\rho \approx 0 \). Strong positive autocorrelation pushes \( d \) towards 0.

Practice Q7. In a study, 30 of 100 exposed and 10 of 100 unexposed subjects develop a condition. A logistic regression of the condition on an exposure indicator \( x \) (1 = exposed) is fitted by maximum likelihood. Which statements are true?

(a) \( \hat\beta_1 = \ln\frac{27}{7} \approx 1.35 \). (b) \( e^{\hat\beta_1} \) is the estimated ratio of the probabilities of the condition, exposed to unexposed. (c) The fitted probability for an exposed subject is 0.3. (d) \( \hat\beta_0 = \ln\frac{1}{9} \).

Answer: (a), (c) and (d)

With one binary predictor the model is saturated (two parameters, two groups), so fitted probabilities equal observed proportions.

  • (a) True. Odds ratio \( \frac{30/70}{10/90} = \frac{27}{7} \approx 3.857 \); its log is about 1.350.
  • (b) False. \( e^{\hat\beta_1} \) is the odds ratio, 3.857. The ratio of probabilities (relative risk) is \( \frac{0.3}{0.1} = 3 \).
  • (c) True. Fitted \( p = 0.3 \) for \( x = 1 \).
  • (d) True. \( \hat\beta_0 \) is the log-odds for \( x = 0 \): \( \ln\frac{0.1}{0.9} = \ln\frac{1}{9} \approx -2.197 \).

Quick Revision Table

IdeaFact
Slope, intercept\( \frac{S_{xy}}{S_{xx}} \), \( \bar y - \hat\beta_1\bar x \)
Slope variance\( \frac{\sigma^2}{S_{xx}} \); error df \( n - 2 \)
\( R^2 \) (simple)\( r^2 = \frac{S_{xy}^2}{S_{xx}S_{yy}} \)
Multiple\( \hat\beta = (X^TX)^{-1}X^TY \), \( \operatorname{Var} = \sigma^2(X^TX)^{-1} \), df \( n - p \)
Adjusted \( R^2 \)\( 1 - (1 - R^2)\frac{n - 1}{n - p} \)
Overall F\( \frac{R^2/(p - 1)}{(1 - R^2)/(n - p)} \)
Leverage\( h_{ii} \in [\frac{1}{n}, 1] \), \( \sum h_{ii} = p \), flag \( > \frac{2p}{n} \)
Residual variance\( \sigma^2(1 - h_{ii}) \)
Cook’s distance\( \frac{r_i^2}{p}\cdot\frac{h_{ii}}{1 - h_{ii}} \)
VIF\( \frac{1}{1 - R_j^2} \)
Durbin–Watson\( \approx 2(1 - \hat\rho) \), range \( [0, 4] \)
Logistic\( \ln\frac{p}{1 - p} = x^T\beta \); \( e^{\beta_j} \) = odds ratio; MLE by IRLS

Key Takeaways

What’s Next