Linear Regression, Covariance and Correlation
CSIR NET Chemical Sciences: least-squares straight lines, slope and intercept errors, linearised plots in kinetics and thermodynamics, covariance and r.
- 22 min
On this page · 14 sections
Prerequisites
- Mean and Standard Deviation: means, deviations and the \( n - 1 \) rule. Regression is built from the same sums.
- Absolute and Relative Errors: uncertainties in slope and intercept propagate into the quantity you extract from them.
- Scatter plots and the idea of positive and negative association from Histograms, Distributions, Scatter and Box Plots.
- The equations being linearised: Beer–Lambert (Spectroscopic Methods of Analysis), integrated rate laws and Arrhenius (Chemical Kinetics), Kohlrausch’s law (Conductance and Kohlrausch’s Law).
The Hook: Six Standards and an Unknown
The Least-Squares Line
We fit \( y = a + bx \) to \( n \) points \( (x_i, y_i) \), assuming:
- the \( x \) values (concentrations of standards, times, temperatures) are known essentially exactly;
- all the error is in \( y \), and it is random with the same SD at every point.
The residual of point \( i \) is the vertical miss \( e_i = y_i - (a + b x_i) \). Least squares chooses \( a \) and \( b \) to minimise
$$S = \sum_{i=1}^{n} e_i^2 = \sum_{i=1}^{n} (y_i - a - b x_i)^2$$Setting \( \partial S/\partial a = 0 \) and \( \partial S/\partial b = 0 \) gives the two normal equations:
$$\sum y_i = n a + b \sum x_i \qquad \sum x_i y_i = a \sum x_i + b \sum x_i^2$$Solving them, with the short-hand sums of squares and products,
$$S_{xx} = \sum (x_i - \bar{x})^2 = \sum x_i^2 - \frac{(\sum x_i)^2}{n} \qquad S_{xy} = \sum (x_i - \bar{x})(y_i - \bar{y}) = \sum x_i y_i - \frac{\sum x_i \sum y_i}{n}$$gives
$$\boxed{b = \frac{S_{xy}}{S_{xx}}} \qquad \boxed{a = \bar{y} - b\,\bar{x}}$$The second result says the least-squares line always passes through the centroid \( (\bar{x}, \bar{y}) \). That is the quickest check on any fitted line, and the fastest way to get the intercept once you have the slope.
Line through the origin
If theory demands \( a = 0 \) (and a blank confirms it), fit \( y = bx \) only. Minimising \( \sum (y_i - b x_i)^2 \) gives
$$b = \frac{\sum x_i y_i}{\sum x_i^2}$$with raw sums, not deviations. For the iron data this gives \( 11.03/220 = 0.05014 \) ppm⁻¹. Force the line through zero only when you have a reason; otherwise a real blank signal is pushed into the slope.
Residuals: how to judge a fit
Plot the residuals \( e_i \) against \( x \).
| Residual pattern | Meaning |
|---|---|
| Random scatter about zero, no trend | straight-line model is adequate |
| Curved (U or arch shape) | the true relation is curved: Beer’s law deviation, wrong kinetic order |
| Fan shape (grow with \( x \)) | error not constant; weighted regression needed |
| One huge residual | possible outlier; check that point |
The residuals of the iron calibration are all within ±0.004 absorbance units with no pattern, as they should be.
Uncertainty in Slope and Intercept
The scatter of points about the line is measured by the standard error of the estimate, with \( n - 2 \) degrees of freedom (two parameters, \( a \) and \( b \), were fitted):
$$\boxed{s_{y/x} = \sqrt{\frac{\sum e_i^2}{n - 2}}}$$From it,
$$\boxed{s_b = \frac{s_{y/x}}{\sqrt{S_{xx}}}} \qquad \boxed{s_a = s_{y/x}\sqrt{\frac{\sum x_i^2}{n\,S_{xx}}}}$$Confidence limits are \( b \pm t s_b \) and \( a \pm t s_a \), with \( t \) for \( n - 2 \) degrees of freedom.
Two practical lessons follow from \( s_b = s_{y/x}/\sqrt{S_{xx}} \):
- Spread the \( x \) values widely. A larger \( S_{xx} \) gives a more precise slope. Standards bunched together give a poor slope however carefully each is measured.
- Predict near the middle. The uncertainty of a concentration read off the line is smallest near \( \bar{x} \) and grows towards the ends. Extrapolating beyond the standards is unreliable.
For one measurement \( y_0 \) of an unknown, the commonly used estimate of the uncertainty in \( x_0 = (y_0 - a)/b \) is
$$s_{x_0} = \frac{s_{y/x}}{b}\sqrt{1 + \frac{1}{n} + \frac{(y_0 - \bar{y})^2}{b^2 S_{xx}}}$$For the unknown above, \( s_{x_0} \approx 0.07 \) ppm. Replicate readings of the unknown shrink the first term (1 becomes \( 1/m \) for \( m \) replicates).
Linearising Chemical Equations
Most laws in physical chemistry are not straight lines as written, but they become straight lines in the right variables. A Part B favourite is “which plot is linear, and what is its slope?”
| Law | Plot \( y \) against \( x \) | Slope | Intercept |
|---|---|---|---|
| Beer–Lambert, \( A = \varepsilon b c \) | \( A \) vs \( c \) | \( \varepsilon b \) | 0 (ideally) |
| Zero order | \( [\text{A}] \) vs \( t \) | \( -k \) | \( [\text{A}]_0 \) |
| First order | \( \ln[\text{A}] \) vs \( t \) | \( -k \) | \( \ln[\text{A}]_0 \) |
| Second order (\( -d[\text{A}]/dt = k[\text{A}]^2 \)) | \( 1/[\text{A}] \) vs \( t \) | \( k \) | \( 1/[\text{A}]_0 \) |
| Arrhenius, \( k = A e^{-E_a/RT} \) | \( \ln k \) vs \( 1/T \) | \( -E_a/R \) | \( \ln A \) |
| Eyring | \( \ln(k/T) \) vs \( 1/T \) | \( -\Delta H^\ddagger/R \) | \( \ln(k_B/h) + \Delta S^\ddagger/R \) |
| van ’t Hoff | \( \ln K \) vs \( 1/T \) | \( -\Delta H^\circ/R \) | \( \Delta S^\circ/R \) |
| Clausius–Clapeyron | \( \ln p \) vs \( 1/T \) | \( -\Delta_{\text{vap}}H/R \) | constant |
| Lineweaver–Burk | \( 1/v \) vs \( 1/[\text{S}] \) | \( K_M/V_{\max} \) | \( 1/V_{\max} \) (x-intercept \( -1/K_M \)) |
| Langmuir | \( p/V \) vs \( p \) | \( 1/V_m \) | \( 1/(K V_m) \) |
| Freundlich | \( \log(x/m) \) vs \( \log p \) | \( 1/n \) | \( \log k \) |
| Stern–Volmer | \( F_0/F \) vs \( [\text{Q}] \) | \( K_{SV} \) | 1 |
| Kohlrausch (strong electrolyte) | \( \Lambda_m \) vs \( \sqrt{c} \) | \( -K \) | \( \Lambda_m^\circ \) |
| Nernst (298 K) | \( E \) vs \( \log Q \) | \( -0.0592/n \) V | \( E^\circ \) |
| Debye–Hückel limiting law | \( \log \gamma_\pm \) vs \( \sqrt{I} \) | \( -A\lvert z_+ z_- \rvert \) | 0 |
Covariance
The covariance measures how two variables vary together:
$$\boxed{\text{cov}(x, y) = s_{xy} = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{n - 1} = \frac{S_{xy}}{n - 1}}$$Each product \( (x_i - \bar{x})(y_i - \bar{y}) \) is positive when both deviations have the same sign (point in the upper-right or lower-left quadrant about the centroid) and negative otherwise.
| Covariance | Pattern |
|---|---|
| Positive | \( y \) tends to rise with \( x \) |
| Negative | \( y \) tends to fall as \( x \) rises |
| Near zero | no linear trend |
Properties:
- \( \text{cov}(x, x) = s_x^2 \), the variance.
- \( \text{cov}(x, y) = \text{cov}(y, x) \).
- \( \text{cov}(px + q,\; uy + v) = p\,u\;\text{cov}(x, y) \): shifts drop out, scale factors multiply.
- Units: units of \( x \) × units of \( y \). Its size depends on the units chosen, so a covariance of 0.696 (ppm × absorbance) by itself says nothing about how strong the relationship is.
Covariance in error propagation
The variance of a sum or difference of two quantities that are not independent is
$$\text{var}(x \pm y) = s_x^2 + s_y^2 \pm 2\,\text{cov}(x, y)$$The quadrature rules on the previous page assume \( \text{cov} = 0 \). The slope and intercept of a fitted line, for example, are correlated (for positive \( \bar{x} \) the covariance of \( a \) and \( b \) is negative), which is why the uncertainty of a value read off a line is not simply \( s_a \) and \( s_b \) combined in quadrature.
The Correlation Coefficient
Dividing the covariance by both SDs removes the units:
$$\boxed{r = \frac{\text{cov}(x, y)}{s_x s_y} = \frac{S_{xy}}{\sqrt{S_{xx} S_{yy}}}} \qquad S_{yy} = \sum y_i^2 - \frac{(\sum y_i)^2}{n}$$This is Pearson’s (product-moment) correlation coefficient.
| Property | Statement |
|---|---|
| Range | \( -1 \le r \le +1 \) |
| \( r = \pm 1 \) | all points exactly on a straight line (rising or falling) |
| \( r = 0 \) | no linear relation (there may be a curved one) |
| Sign | same as the sign of the slope and of the covariance |
| Units | none; unchanged by change of units or origin |
| Under \( u = px + q \), \( v = cy + d \) | \( r_{uv} = r_{xy} \) if \( p \) and \( c \) have the same sign; \( -r_{xy} \) if opposite |
| Symmetry | \( r_{xy} = r_{yx} \) |
Links between \( r \) and the regression slope
$$\boxed{b_{yx} = r\,\frac{s_y}{s_x}} \qquad \boxed{b_{xy} = r\,\frac{s_x}{s_y}} \qquad \boxed{b_{yx}\, b_{xy} = r^2}$$Here \( b_{yx} \) is the slope of the regression of \( y \) on \( x \) (minimising vertical misses), and \( b_{xy} \) is the slope of \( x \) on \( y \) (minimising horizontal misses).
- The two regression lines are different unless \( r = \pm 1 \), and both pass through \( (\bar{x}, \bar{y}) \). So the means are found by solving the two line equations together.
- \( b_{yx} \), \( b_{xy} \) and \( r \) all have the same sign.
- Because \( r^2 \le 1 \), the product of the two slopes cannot exceed 1. This tells you which given equation is \( y \) on \( x \): try both assignments and keep the one whose slope product is at most 1.
The coefficient of determination
\( r^2 \) is the fraction of the variation in \( y \) that the straight line accounts for:
$$r^2 = 1 - \frac{\sum e_i^2}{S_{yy}}$$\( r = 0.9 \) means \( r^2 = 0.81 \): the line explains 81% of the variance in \( y \), not 90%.
What \( r \) cannot tell you
Memory Tricks
Don’t Confuse
| This | vs | That |
|---|---|---|
| Regression: predicts \( y \) from \( x \) (direction matters) | correlation: symmetric measure of linear association | |
| Covariance: has units, depends on scale | \( r \): dimensionless, scale-free | |
| \( r \) = 0.9 | \( r^2 \) = 0.81 (fraction of variance explained) | |
| Regression of \( y \) on \( x \): slope \( r s_y/s_x \) | of \( x \) on \( y \): slope \( r s_x/s_y \) (plotted as \( y \) vs \( x \), its slope is \( 1/b_{xy} \)) | |
| \( s_{y/x} \) with \( n - 2 \) | SD of replicates with \( n - 1 \) | |
| \( r = 0 \): no linear relation | independence (stronger; \( y = x^2 \) has \( r = 0 \)) | |
| Slope of \( \ln k \) vs \( 1/T \) is \( -E_a/R \) | slope of \( \log_{10} k \) vs \( 1/T \) is \( -E_a/(2.303R) \) |
Common Mistakes
Practice
All questions below are original practice questions written for this page. They are not from any CSIR NET paper.
Level 1: Foundation
- Through which point does every least-squares line \( y = a + bx \) pass?
- Between what limits does the correlation coefficient lie, and what does \( r = -1 \) mean?
- What should be plotted to get a straight line for a second-order reaction (\( -d[\text{A}]/dt = k[\text{A}]^2 \)), and what is the slope?
Answers
- The centroid \( (\bar{x}, \bar{y}) \).
- \( -1 \le r \le 1 \). \( r = -1 \): all points lie exactly on a straight line of negative slope.
- \( 1/[\text{A}] \) against \( t \); slope \( k \), intercept \( 1/[\text{A}]_0 \).
Level 2: Part B Style
Practice Q1. For five data points, \( \sum x = 15 \), \( \sum y = 20 \), \( \sum x^2 = 55 \) and \( \sum xy = 66 \). The least-squares line of \( y \) on \( x \) is:
(a) \( y = 2.2 + 0.6x \) (b) \( y = 1.6 + 0.8x \) (c) \( y = 0.6 + 2.2x \) (d) \( y = 4.0 + 0.6x \)
Answer: (a)
\( S_{xy} = 66 - (15)(20)/5 = 6 \); \( S_{xx} = 55 - 15^2/5 = 10 \); \( b = 0.6 \).
\( a = \bar{y} - b\bar{x} = 4 - 0.6 \times 3 = 2.2 \).
Check: the line passes through (3, 4). (d) uses \( \bar{y} \) as the intercept and fails this check.
Practice Q2. For the points \( (-2, 4), (-1, 1), (0, 0), (1, 1), (2, 4) \), the correlation coefficient is:
(a) +1 (b) −1 (c) 0 (d) +0.5
Answer: (c) 0
\( \bar{x} = 0 \), so \( S_{xy} = \sum x_i y_i = -8 - 1 + 0 + 1 + 8 = 0 \), hence \( r = 0 \).
The data lie exactly on \( y = x^2 \). \( r = 0 \) means no linear relation, not no relation.
Practice Q3. The correlation coefficient between \( x \) and \( y \) is 0.8. If \( u = 2x + 3 \) and \( v = 5 - 3y \), the correlation coefficient between \( u \) and \( v \) is:
(a) 0.8 (b) −0.8 (c) −4.8 (d) −0.53
Answer: (b) −0.8
Shifts do not affect \( r \); the scale factors 2 and −3 have opposite signs, so the sign flips and the size is unchanged: \( r_{uv} = -0.8 \).
(c) multiplies by \( 2 \times (-3) \), which is what happens to the covariance, not to \( r \).
Practice Q4. For a reaction, a plot of \( \ln[\text{A}] \) against \( t \) (in minutes) is a straight line with slope \( -0.0300 \) min⁻¹. The half-life is about:
(a) 10.0 min (b) 16.7 min (c) 23.1 min (d) 33.3 min
Answer: (c) 23.1 min
A linear \( \ln[\text{A}] \) vs \( t \) plot means first order, with \( k = 0.0300 \) min⁻¹.
\( t_{1/2} = \ln 2/k = 0.693/0.0300 = 23.1 \) min.
(d) is \( 1/k \), the mean lifetime, not the half-life.
Level 3: Part C Style
Practice Q5. Rate constants for a reaction at four temperatures:
| \( T \) / K | 300 | 310 | 330 | 340 |
|---|---|---|---|---|
| \( 10^3/T \) / K⁻¹ | 3.3333 | 3.2258 | 3.0303 | 2.9412 |
| \( \ln(k/\text{s}^{-1}) \) | −3.333 | −2.556 | −1.146 | −0.503 |
Take \( R = 8.314 \) J K⁻¹ mol⁻¹. The activation energy is closest to:
(a) 26 kJ mol⁻¹ (b) 60 kJ mol⁻¹ (c) 0.060 kJ mol⁻¹ (d) 138 kJ mol⁻¹
Answer: (b) 60 kJ mol⁻¹
The points are collinear, so take the slope from the end points (a least-squares fit gives the same value):
\( \text{slope} = \frac{-0.503 - (-3.333)}{(2.9412 - 3.3333) \times 10^{-3}} = \frac{2.830}{-3.921 \times 10^{-4}} = -7.22 \times 10^{3} \) K.
\( E_a = -R \times \text{slope} = 8.314 \times 7217 = 6.00 \times 10^{4} \) J mol⁻¹ = 60.0 kJ mol⁻¹.
(a) divides by 2.303 and (d) multiplies by 2.303: both confuse natural and base-10 logs, although the data are natural logs. (c) forgets that the axis is \( 10^3/T \), so the slope read off it (−7.22) is 1000 times too small.
Practice Q6. For a set of paired data, the regression line of \( y \) on \( x \) is \( y = 0.9x + 2.4 \) and the regression line of \( x \) on \( y \) is \( x = 0.4y + 1.6 \). Consider the statements:
- \( \bar{x} = 4 \) and \( \bar{y} = 6 \).
- The correlation coefficient is +0.6.
- The correlation coefficient is 0.36.
- If \( s_x = 2 \), then \( s_y = 3 \).
Which statements are correct?
(a) 1 and 2 only (b) 1, 2 and 4 only (c) 1 and 3 only (d) 2 and 4 only
Answer: (b) 1, 2 and 4 only
- Both lines pass through \( (\bar{x}, \bar{y}) \). Substitute: \( x = 0.4(0.9x + 2.4) + 1.6 = 0.36x + 2.56 \Rightarrow 0.64x = 2.56 \Rightarrow \bar{x} = 4 \); \( \bar{y} = 0.9 \times 4 + 2.4 = 6 \). Correct.
- \( r^2 = b_{yx} b_{xy} = 0.9 \times 0.4 = 0.36 \); both slopes positive, so \( r = +0.6 \). Correct.
- 0.36 is \( r^2 \), not \( r \). Incorrect.
- \( b_{yx} = r s_y/s_x \Rightarrow s_y = 0.9 \times 2/0.6 = 3 \). Correct.
Check on the assignment: had the lines been swapped, the slopes would be \( 1/0.4 = 2.5 \) and \( 1/0.9 = 1.11 \), with product 2.78 > 1, which is impossible. So the labelling given is consistent.
Practice Q7. A fluorescence calibration with five standards gives \( n = 5 \), \( \sum x = 15 \) µM, \( \sum y = 160.5 \), \( \sum x^2 = 55 \) µM², \( \sum xy = 581.2 \) µM (\( y \) = intensity in arbitrary units). An unknown gives an intensity of 37.0. Its concentration is closest to:
(a) 3.49 µM (b) 3.71 µM (c) 3.21 µM (d) 4.03 µM
Answer: (a) 3.49 µM
\( S_{xx} = 55 - 15^2/5 = 10 \); \( S_{xy} = 581.2 - (15 \times 160.5)/5 = 581.2 - 481.5 = 99.7 \).
\( b = 99.7/10 = 9.97 \) µM⁻¹; \( a = 32.1 - 9.97 \times 3 = 2.19 \).
\( x_0 = (37.0 - 2.19)/9.97 = 3.49 \) µM.
(b) ignores the intercept: \( 37.0/9.97 = 3.71 \) µM. The unknown lies close to \( \bar{x} = 3 \) µM, the most precise region of the line.
Quick Revision Table
| Quantity | Formula or fact |
|---|---|
| \( S_{xx} \), \( S_{yy} \), \( S_{xy} \) | \( \sum x^2 - (\sum x)^2/n \), \( \sum y^2 - (\sum y)^2/n \), \( \sum xy - \sum x \sum y/n \) |
| Slope, intercept | \( b = S_{xy}/S_{xx} \); \( a = \bar{y} - b\bar{x} \) |
| Through origin | \( b = \sum xy/\sum x^2 \) |
| Scatter about line | \( s_{y/x} = \sqrt{\sum e_i^2/(n - 2)} \) |
| Slope, intercept SDs | \( s_b = s_{y/x}/\sqrt{S_{xx}} \); \( s_a = s_{y/x}\sqrt{\sum x^2/(n S_{xx})} \) |
| Unknown | \( x_0 = (y_0 - a)/b \) |
| Covariance | \( S_{xy}/(n - 1) \); units of \( x \) × \( y \) |
| Correlation | \( r = S_{xy}/\sqrt{S_{xx}S_{yy}} \), \( -1 \le r \le 1 \) |
| Slopes and r | \( b_{yx} = r s_y/s_x \); \( b_{xy} = r s_x/s_y \); \( b_{yx} b_{xy} = r^2 \) |
| \( r^2 \) | fraction of variance in \( y \) explained |
| Arrhenius plot | \( \ln k \) vs \( 1/T \): slope \( -E_a/R \) |
| First / second order | \( \ln[\text{A}] \) vs \( t \): \( -k \); \( 1/[\text{A}] \) vs \( t \): \( k \) |
Key Takeaways
What’s Next
- Data Analysis: Quick Revision: all three pages on one sheet.
- Spectroscopic Methods of Analysis: calibration lines, detection limits and standard addition in use.
- Conductance and Kohlrausch’s Law: a linear extrapolation to infinite dilution.
- Mathematical Sciences: regression and correlation from the statistics side, for candidates who want the theory.