Partial and Multiple Correlation
CSIR NET statistics: partial and multiple correlation coefficients, three-variable formulas, the inverse correlation matrix, bounds, and t and F tests.
- 16 min
On this page · 15 sections
Prerequisites
- Descriptive Statistics and Exploratory Data Analysis: the simple correlation \( r \) and the two regression lines.
- Simple, Multiple and Logistic Regression: \( R^2 \), residuals and the F test.
- Multivariate Normal, Wishart and Quadratic Forms: the conditional covariance \( \Sigma_{11\cdot2} \) and the precision matrix.
- Quadratic Forms: a correlation matrix must be positive semidefinite.
The Hook: Ice Cream and Drowning
Notation
For variables \( X_1, \dots, X_p \) with correlations \( r_{ij} \) (sample) or \( \rho_{ij} \) (population):
- \( r_{12\cdot3} \): partial correlation between \( X_1 \) and \( X_2 \), eliminating \( X_3 \). The variables after the dot are held fixed (removed). The number of variables after the dot is the order: \( r_{12} \) is zero order, \( r_{12\cdot3} \) first order, \( r_{12\cdot34} \) second order.
- \( R_{1\cdot23} \): multiple correlation of \( X_1 \) on \( X_2 \) and \( X_3 \). The variable before the dot is predicted, those after are predictors.
Partial Correlation
Let \( e_{1\cdot3} \) be the residual of \( X_1 \) after its least squares linear regression on \( X_3 \), and \( e_{2\cdot3} \) the residual of \( X_2 \) on \( X_3 \). Then
$$r_{12\cdot3} = \operatorname{corr}(e_{1\cdot3}, e_{2\cdot3}).$$More generally, \( r_{ij\cdot S} \) is the correlation of the residuals of \( X_i \) and \( X_j \) after regression on the set \( S \).
Working out the residual covariance gives the three-variable formula:
$$\boxed{r_{12\cdot3} = \frac{r_{12} - r_{13}r_{23}}{\sqrt{(1 - r_{13}^2)(1 - r_{23}^2)}}}$$and the recursion for higher orders, peeling off one variable at a time:
$$r_{12\cdot34} = \frac{r_{12\cdot3} - r_{14\cdot3}r_{24\cdot3}}{\sqrt{(1 - r_{14\cdot3}^2)(1 - r_{24\cdot3}^2)}}.$$Reading the formula.
- \( r_{12\cdot3} = 0 \) iff \( r_{12} = r_{13}r_{23} \): the whole association of \( X_1 \) and \( X_2 \) runs through \( X_3 \).
- \( r_{12\cdot3} \) can have the opposite sign to \( r_{12} \), and can be larger in magnitude.
- If \( r_{13} = 0 \) or \( r_{23} = 0 \), the numerator is \( r_{12} \) but the denominator is still \( < 1 \) in general, so \( \lvert r_{12\cdot3} \rvert \ge \lvert r_{12} \rvert \).
- \( -1 \le r_{12\cdot3} \le 1 \), like any correlation.
Population meaning for normal data
For a jointly normal vector, the population partial correlation \( \rho_{12\cdot3} \) is the correlation in the conditional distribution of \( (X_1, X_2) \) given \( X_3 = x_3 \), and it does not depend on \( x_3 \). So for normal data
$$\rho_{12\cdot3} = 0 \iff X_1 \perp X_2 \mid X_3.$$For non-normal data a zero partial correlation only says the residuals of the linear regressions are uncorrelated.
Multiple Correlation
Three-variable formula:
$$\boxed{R^2_{1\cdot23} = \frac{r_{12}^2 + r_{13}^2 - 2r_{12}r_{13}r_{23}}{1 - r_{23}^2}}$$and the product identity linking it to partial correlations:
$$1 - R^2_{1\cdot23} = (1 - r_{12}^2)(1 - r_{13\cdot2}^2), \qquad 1 - R^2_{1\cdot23\ldots p} = (1 - r_{12}^2)(1 - r_{13\cdot2}^2)(1 - r_{14\cdot23}^2)\cdots.$$Each factor is the proportion of variance left unexplained after adding one more predictor. The residual (partial) variance is \( \sigma^2_{1\cdot23\ldots p} = \sigma_1^2(1 - R^2_{1\cdot23\ldots p}) \).
Properties.
| Property | Statement |
|---|---|
| range | \( 0 \le R \le 1 \) (never negative) |
| dominates simple correlations | \( R_{1\cdot23} \ge \max(\lvert r_{12} \rvert, \lvert r_{13} \rvert) \) |
| monotone | adding predictors never decreases \( R \) |
| zero | \( R_{1\cdot23} = 0 \iff r_{12} = r_{13} = 0 \) |
| one | \( R_{1\cdot23} = 1 \iff X_1 \) is exactly a linear function of \( X_2, X_3 \) |
| uncorrelated predictors | if \( r_{23} = 0 \), \( R^2_{1\cdot23} = r_{12}^2 + r_{13}^2 \) |
| asymmetric | \( R_{1\cdot23} \ne R_{2\cdot13} \) in general |
Everything From the Inverse Correlation Matrix
Let \( R = (r_{ij}) \) be the \( p \times p \) correlation matrix (positive definite) and \( P = R^{-1} = (p^{ij}) \). Then
$$\boxed{r_{ij\cdot\text{rest}} = -\frac{p^{ij}}{\sqrt{p^{ii}p^{jj}}}, \qquad R^2_{i\cdot\text{rest}} = 1 - \frac{1}{p^{ii}} = 1 - \frac{\lvert R \rvert}{\lvert R_{ii} \rvert},}$$where “rest” means all the other variables and \( \lvert R_{ii} \rvert \) is the minor obtained by deleting row and column \( i \). With the covariance matrix instead, \( \Sigma^{-1} = (\sigma^{ij}) \), the partial correlation formula is unchanged (the scales cancel), while \( R^2_{i\cdot\text{rest}} = 1 - \frac{1}{\sigma_{ii}\sigma^{ii}} \). For the diagonal: \( \frac{1}{p^{ii}} = 1 - R^2_{i\cdot\text{rest}} \), and \( p^{ii} \) is the variance inflation factor of \( X_i \) when all the other variables are used as predictors together with it.
Which Correlations Are Possible?
A correlation matrix must be positive semidefinite. For three variables this is
$$\lvert R \rvert = 1 - r_{12}^2 - r_{13}^2 - r_{23}^2 + 2r_{12}r_{13}r_{23} \ge 0,$$which, for given \( r_{12} \) and \( r_{13} \), confines \( r_{23} \) to
$$r_{12}r_{13} - \sqrt{(1 - r_{12}^2)(1 - r_{13}^2)} \;\le\; r_{23} \;\le\; r_{12}r_{13} + \sqrt{(1 - r_{12}^2)(1 - r_{13}^2)}.$$Example: \( r_{12} = r_{13} = 0.9 \) forces \( 0.62 \le r_{23} \le 1 \). Correlation is not transitive in general (with \( r_{12} = r_{13} = 0.5 \), \( r_{23} \) can be anything from \( -0.5 \) to 1), but strong correlations force a positive one.
Equicorrelation. If every \( r_{ij} = \rho \) among three variables: \( r_{12\cdot3} = \frac{\rho}{1 + \rho} \), \( R^2_{1\cdot23} = \frac{2\rho^2}{1 + \rho} \), and \( \lvert R \rvert = (1 - \rho)^2(1 + 2\rho) \), which needs \( \rho \ge -\frac{1}{2} \).
Tests
For a sample of size \( n \) from a multivariate normal population:
| Hypothesis | Statistic | Null distribution |
|---|---|---|
| \( \rho_{12} = 0 \) | \( t = \frac{r\sqrt{n - 2}}{\sqrt{1 - r^2}} \) | \( t_{n - 2} \) |
| \( \rho_{12\cdot S} = 0 \), \( k \) variables in \( S \) | \( t = \frac{r_{12\cdot S}\sqrt{n - 2 - k}}{\sqrt{1 - r_{12\cdot S}^2}} \) | \( t_{n - 2 - k} \) |
| \( R_{1\cdot2\ldots(k+1)} = 0 \) (\( k \) predictors) | \( F = \frac{R^2/k}{(1 - R^2)/(n - k - 1)} \) | \( F_{k, n - k - 1} \) |
A partial correlation of order \( k \) behaves like a simple correlation computed from \( n - k \) observations: each variable partialled out costs one degree of freedom. The multiple correlation test is the overall regression F test.
graph TD
A["Given r12, r13, r23"] --> B{"What is asked?"}
B -->|"partial r12.3"| C["(r12 - r13 r23) / sqrt((1 - r13^2)(1 - r23^2))"]
B -->|"multiple R1.23"| D["R^2 = (r12^2 + r13^2 - 2 r12 r13 r23) / (1 - r23^2)"]
B -->|"many variables"| E["Invert R: partial = -p_ij / sqrt(p_ii p_jj),
R^2 = 1 - 1/p_ii"]
B -->|"is it possible?"| F["Check det R >= 0"]Memory Tricks
- Partial: “subtract the path through 3, then rescale.” Numerator \( r_{12} - r_{13}r_{23} \); denominator uses the two correlations with 3.
- Multiple: “sum of squares minus twice the triple product, over one minus the predictors’ r².”
- Precision matrix: minus sign for partials, reciprocal for R². \( -\frac{p^{ij}}{\sqrt{p^{ii}p^{jj}}} \) and \( 1 - \frac{1}{p^{ii}} \).
- Each partialled variable costs one df. \( t_{n - 2 - k} \).
- R never loses. \( R_{1\cdot23} \ge \lvert r_{12} \rvert \), \( \lvert r_{13} \rvert \), and it is never negative.
Don’t Confuse
| This | Is not this | The difference |
|---|---|---|
| \( r_{12\cdot3} \) (partial) | \( R_{1\cdot23} \) (multiple) | partial: two variables, a third removed; multiple: one variable against a set |
| \( R_{1\cdot23} \) | \( R_{2\cdot13} \) | different dependent variable, different value |
| partial correlation \( = 0 \) (normal data) | partial correlation \( = 0 \) (any data) | conditional independence only under normality |
| \( r_{12\cdot3} \) with sign of \( r_{12} \) | \( r_{12\cdot3} \) | it can reverse sign |
| multiple correlation \( R \) | simple correlation \( r \) | \( 0 \le R \le 1 \) always, while \( r \in [-1, 1] \); \( R \) is the correlation with the best linear predictor, whose sign is chosen to make it non-negative |
| \( \Sigma^{-1} \) entries | \( \Sigma \) entries | the inverse gives partials and \( R^2 \); \( \Sigma \) gives simple covariances |
| partial correlation | semi-partial (part) correlation | the part correlation removes \( X_3 \) from only one of the two variables |
Common Mistakes
Practice
All questions below are original practice questions written for this page. They are not from any CSIR NET paper. In Level 3, as in Part C, one or more options may be correct.
Level 1: Foundation
- Write \( r_{12\cdot3} \) in terms of the simple correlations.
- What does \( R_{1\cdot23} = 0 \) say about \( r_{12} \) and \( r_{13} \)?
- If \( r_{12} = r_{13}r_{23} \), what is \( r_{12\cdot3} \)?
Answers
- \( \frac{r_{12} - r_{13}r_{23}}{\sqrt{(1 - r_{13}^2)(1 - r_{23}^2)}} \).
- Both are zero.
- Zero (the numerator vanishes).
Level 2: Part B Style
Practice Q1. Given \( r_{12} = 0.6 \), \( r_{13} = 0.5 \), \( r_{23} = 0.4 \), the partial correlation \( r_{12\cdot3} \) is closest to:
(a) 0.40 (b) 0.50 (c) 0.56 (d) 0.66
Answer: (b)
\( \frac{0.6 - 0.2}{\sqrt{0.75 \times 0.84}} = \frac{0.4}{0.794} \approx 0.504 \). Option (a) forgets the denominator; (d) is \( R_{1\cdot23} \).
Practice Q2. With the same correlations, \( R^2_{1\cdot23} \) is closest to:
(a) 0.36 (b) 0.44 (c) 0.61 (d) 0.66
Answer: (b)
\( \frac{0.36 + 0.25 - 2(0.6)(0.5)(0.4)}{1 - 0.16} = \frac{0.37}{0.84} \approx 0.440 \). Option (c) is \( r_{12}^2 + r_{13}^2 \), which ignores the correlation between the predictors.
Practice Q3. Given \( r_{12} = 0.3 \), \( r_{13} = 0.8 \), \( r_{23} = 0.6 \), the partial correlation \( r_{12\cdot3} \) is:
(a) 0.375 (b) \( -0.375 \) (c) \( -0.18 \) (d) 0.3
Answer: (b)
\( \frac{0.3 - 0.48}{\sqrt{(1 - 0.64)(1 - 0.36)}} = \frac{-0.18}{0.6 \times 0.8} = -0.375 \). The sign reverses once \( X_3 \) is held fixed.
Practice Q4. From a normal sample of \( n = 30 \), the first-order partial correlation \( r_{12\cdot3} = 0.5 \). The t statistic for \( H_0: \rho_{12\cdot3} = 0 \) and its degrees of freedom are:
(a) 3.0 on 27 df (b) 3.06 on 28 df (c) 2.6 on 27 df (d) 3.0 on 28 df
Answer: (a)
\( t = \frac{0.5\sqrt{30 - 2 - 1}}{\sqrt{1 - 0.25}} = \frac{0.5\sqrt{27}}{\sqrt{0.75}} = 0.5\sqrt{36} = 3.0 \), on \( 27 \) df. Option (b) treats it as a simple correlation, with \( n - 2 = 28 \) df.
Level 3: Part C Style
Practice Q5. Three variables have all pairwise correlations equal to \( 0.5 \). Which statements are true?
(a) \( r_{12\cdot3} = \frac{1}{3} \). (b) \( R^2_{1\cdot23} = \frac{1}{3} \). (c) \( R_{1\cdot23} < r_{12} \). (d) The correlation matrix is singular.
Answer: (a) and (b)
- (a) True. \( \frac{0.5 - 0.25}{1 - 0.25} = \frac{1}{3} \) (generally \( \frac{\rho}{1 + \rho} \)).
- (b) True. \( \frac{0.25 + 0.25 - 0.25}{0.75} = \frac{1}{3} \) (generally \( \frac{2\rho^2}{1 + \rho} \)).
- (c) False. \( R_{1\cdot23} = \frac{1}{\sqrt{3}} \approx 0.577 > 0.5 \); a multiple correlation is never below a simple one with the same dependent variable.
- (d) False. \( \lvert R \rvert = (1 - 0.5)^2(1 + 1) = 0.5 \ne 0 \).
Practice Q6. The inverse of the correlation matrix of \( (X_1, X_2, X_3) \) is \( P = \frac{1}{47}\begin{pmatrix} 84 & -40 & -26 \\ -40 & 75 & -10 \\ -26 & -10 & 64 \end{pmatrix} \). Which statements are true?
(a) \( R^2_{1\cdot23} = \frac{37}{84} \). (b) \( r_{12\cdot3} = \frac{40}{\sqrt{84 \times 75}} \approx 0.504 \). (c) \( r_{12\cdot3} \) is negative because \( p^{12} \) is negative. (d) The variance inflation factor of \( X_1 \) in the regression of \( X_3 \) on \( X_1, X_2 \) is \( \frac{84}{47} \).
Answer: (a) and (b)
(This \( P \) is the inverse of the matrix with \( r_{12} = 0.6 \), \( r_{13} = 0.5 \), \( r_{23} = 0.4 \).)
- (a) True. \( 1 - \frac{1}{p^{11}} = 1 - \frac{47}{84} = \frac{37}{84} \approx 0.440 \).
- (b) True. \( -\frac{p^{12}}{\sqrt{p^{11}p^{22}}} = \frac{40/47}{\sqrt{84 \times 75}/47} \).
- (c) False. The formula carries a minus sign: a negative off-diagonal entry of the precision matrix means a positive partial correlation.
- (d) False. \( p^{11} = \frac{84}{47} \) is the VIF of \( X_1 \) when all the other variables, \( X_2 \) and \( X_3 \), are its fellow predictors. In the regression of \( X_3 \) on \( X_1, X_2 \), the only other predictor is \( X_2 \), so the VIF of \( X_1 \) is \( \frac{1}{1 - r_{12}^2} = \frac{1}{0.64} = 1.5625 \).
Practice Q7. Suppose \( r_{12} = r_{13} = 0.9 \). Which statements are true?
(a) \( r_{23} = 0 \) is possible. (b) \( r_{23} \) must be at least 0.62. (c) If \( r_{23} = 0.81 \), then \( r_{23\cdot1} = 0 \). (d) If \( r_{23} = 0.81 \), then \( R^2_{1\cdot23} > 0.81 \).
Answer: (b), (c) and (d)
- (a) False. \( \lvert R \rvert = 1 - 0.81 - 0.81 = -0.62 < 0 \).
- (b) True. \( r_{23} \ge 0.81 - \sqrt{0.19 \times 0.19} = 0.62 \).
- (c) True. \( r_{23\cdot1} = \frac{0.81 - 0.9 \times 0.9}{0.19} = 0 \): \( X_2 \) and \( X_3 \) are related only through \( X_1 \).
- (d) True. \( R^2_{1\cdot23} = \frac{0.81 + 0.81 - 2(0.81)(0.81)}{1 - 0.6561} = \frac{0.3078}{0.3439} \approx 0.895 \). It must be at least \( r_{12}^2 = 0.81 \), and here it is strictly larger.
Quick Revision Table
| Idea | Fact |
|---|---|
| Partial (definition) | correlation of residuals after linear regression on the removed variables |
| \( r_{12\cdot3} \) | \( \frac{r_{12} - r_{13}r_{23}}{\sqrt{(1 - r_{13}^2)(1 - r_{23}^2)}} \) |
| Recursion | \( r_{12\cdot34} \) from first-order partials with 4 removed |
| \( R^2_{1\cdot23} \) | \( \frac{r_{12}^2 + r_{13}^2 - 2r_{12}r_{13}r_{23}}{1 - r_{23}^2} \) |
| Product identity | \( 1 - R^2_{1\cdot23} = (1 - r_{12}^2)(1 - r_{13\cdot2}^2) \) |
| From \( P = R^{-1} \) | \( r_{ij\cdot\text{rest}} = -\frac{p^{ij}}{\sqrt{p^{ii}p^{jj}}} \), \( R^2_{i\cdot\text{rest}} = 1 - \frac{1}{p^{ii}} \) |
| Bounds | \( 0 \le R \le 1 \); \( R_{1\cdot23} \ge \lvert r_{12} \rvert, \lvert r_{13} \rvert \) |
| Feasibility | \( 1 - r_{12}^2 - r_{13}^2 - r_{23}^2 + 2r_{12}r_{13}r_{23} \ge 0 \) |
| Equicorrelation | \( r_{12\cdot3} = \frac{\rho}{1 + \rho} \), \( R^2_{1\cdot23} = \frac{2\rho^2}{1 + \rho} \) |
| Tests | partial of order \( k \): \( t_{n - 2 - k} \); multiple: \( F_{k, n - k - 1} \) |
| Normal data | \( \rho_{12\cdot3} = 0 \iff X_1 \perp X_2 \mid X_3 \) |
Key Takeaways
What’s Next
- PCA, Discriminant, Cluster and Canonical Correlation Analysis: canonical correlation generalises multiple correlation to two sets of variables.
- Simple, Multiple and Logistic Regression: \( R^2 \), VIF and the F test in regression form.
- Multivariate Normal, Wishart and Quadratic Forms: conditional covariance and the precision matrix.
- Back to the Linear Models and Multivariate Analysis chapter overview.