Skip to content

Linear Regression, Covariance and Correlation

CSIR NET Chemical Sciences: least-squares straight lines, slope and intercept errors, linearised plots in kinetics and thermodynamics, covariance and r.

  • 22 min read
  • Updated
On this page · 14 sections
  1. Prerequisites
  2. The Hook: Six Standards and an Unknown
  3. The Least-Squares Line
  4. Uncertainty in Slope and Intercept
  5. Linearising Chemical Equations
  6. Covariance
  7. The Correlation Coefficient
  8. Memory Tricks
  9. Don’t Confuse
  10. Common Mistakes
  11. Practice
  12. Quick Revision Table
  13. Key Takeaways
  14. What’s Next

Prerequisites

The Hook: Six Standards and an Unknown

The Least-Squares Line

We fit \( y = a + bx \) to \( n \) points \( (x_i, y_i) \), assuming:

  1. the \( x \) values (concentrations of standards, times, temperatures) are known essentially exactly;
  2. all the error is in \( y \), and it is random with the same SD at every point.

The residual of point \( i \) is the vertical miss \( e_i = y_i - (a + b x_i) \). Least squares chooses \( a \) and \( b \) to minimise

$$S = \sum_{i=1}^{n} e_i^2 = \sum_{i=1}^{n} (y_i - a - b x_i)^2$$

Setting \( \partial S/\partial a = 0 \) and \( \partial S/\partial b = 0 \) gives the two normal equations:

$$\sum y_i = n a + b \sum x_i \qquad \sum x_i y_i = a \sum x_i + b \sum x_i^2$$

Solving them, with the short-hand sums of squares and products,

$$S_{xx} = \sum (x_i - \bar{x})^2 = \sum x_i^2 - \frac{(\sum x_i)^2}{n} \qquad S_{xy} = \sum (x_i - \bar{x})(y_i - \bar{y}) = \sum x_i y_i - \frac{\sum x_i \sum y_i}{n}$$

gives

$$\boxed{b = \frac{S_{xy}}{S_{xx}}} \qquad \boxed{a = \bar{y} - b\,\bar{x}}$$

The second result says the least-squares line always passes through the centroid \( (\bar{x}, \bar{y}) \). That is the quickest check on any fitted line, and the fastest way to get the intercept once you have the slope.

Line through the origin

If theory demands \( a = 0 \) (and a blank confirms it), fit \( y = bx \) only. Minimising \( \sum (y_i - b x_i)^2 \) gives

$$b = \frac{\sum x_i y_i}{\sum x_i^2}$$

with raw sums, not deviations. For the iron data this gives \( 11.03/220 = 0.05014 \) ppm⁻¹. Force the line through zero only when you have a reason; otherwise a real blank signal is pushed into the slope.

Residuals: how to judge a fit

Plot the residuals \( e_i \) against \( x \).

Residual patternMeaning
Random scatter about zero, no trendstraight-line model is adequate
Curved (U or arch shape)the true relation is curved: Beer’s law deviation, wrong kinetic order
Fan shape (grow with \( x \))error not constant; weighted regression needed
One huge residualpossible outlier; check that point

The residuals of the iron calibration are all within ±0.004 absorbance units with no pattern, as they should be.

Uncertainty in Slope and Intercept

The scatter of points about the line is measured by the standard error of the estimate, with \( n - 2 \) degrees of freedom (two parameters, \( a \) and \( b \), were fitted):

$$\boxed{s_{y/x} = \sqrt{\frac{\sum e_i^2}{n - 2}}}$$

From it,

$$\boxed{s_b = \frac{s_{y/x}}{\sqrt{S_{xx}}}} \qquad \boxed{s_a = s_{y/x}\sqrt{\frac{\sum x_i^2}{n\,S_{xx}}}}$$

Confidence limits are \( b \pm t s_b \) and \( a \pm t s_a \), with \( t \) for \( n - 2 \) degrees of freedom.

Two practical lessons follow from \( s_b = s_{y/x}/\sqrt{S_{xx}} \):

  • Spread the \( x \) values widely. A larger \( S_{xx} \) gives a more precise slope. Standards bunched together give a poor slope however carefully each is measured.
  • Predict near the middle. The uncertainty of a concentration read off the line is smallest near \( \bar{x} \) and grows towards the ends. Extrapolating beyond the standards is unreliable.

For one measurement \( y_0 \) of an unknown, the commonly used estimate of the uncertainty in \( x_0 = (y_0 - a)/b \) is

$$s_{x_0} = \frac{s_{y/x}}{b}\sqrt{1 + \frac{1}{n} + \frac{(y_0 - \bar{y})^2}{b^2 S_{xx}}}$$

For the unknown above, \( s_{x_0} \approx 0.07 \) ppm. Replicate readings of the unknown shrink the first term (1 becomes \( 1/m \) for \( m \) replicates).

Linearising Chemical Equations

Most laws in physical chemistry are not straight lines as written, but they become straight lines in the right variables. A Part B favourite is “which plot is linear, and what is its slope?”

LawPlot \( y \) against \( x \)SlopeIntercept
Beer–Lambert, \( A = \varepsilon b c \)\( A \) vs \( c \)\( \varepsilon b \)0 (ideally)
Zero order\( [\text{A}] \) vs \( t \)\( -k \)\( [\text{A}]_0 \)
First order\( \ln[\text{A}] \) vs \( t \)\( -k \)\( \ln[\text{A}]_0 \)
Second order (\( -d[\text{A}]/dt = k[\text{A}]^2 \))\( 1/[\text{A}] \) vs \( t \)\( k \)\( 1/[\text{A}]_0 \)
Arrhenius, \( k = A e^{-E_a/RT} \)\( \ln k \) vs \( 1/T \)\( -E_a/R \)\( \ln A \)
Eyring\( \ln(k/T) \) vs \( 1/T \)\( -\Delta H^\ddagger/R \)\( \ln(k_B/h) + \Delta S^\ddagger/R \)
van ’t Hoff\( \ln K \) vs \( 1/T \)\( -\Delta H^\circ/R \)\( \Delta S^\circ/R \)
Clausius–Clapeyron\( \ln p \) vs \( 1/T \)\( -\Delta_{\text{vap}}H/R \)constant
Lineweaver–Burk\( 1/v \) vs \( 1/[\text{S}] \)\( K_M/V_{\max} \)\( 1/V_{\max} \) (x-intercept \( -1/K_M \))
Langmuir\( p/V \) vs \( p \)\( 1/V_m \)\( 1/(K V_m) \)
Freundlich\( \log(x/m) \) vs \( \log p \)\( 1/n \)\( \log k \)
Stern–Volmer\( F_0/F \) vs \( [\text{Q}] \)\( K_{SV} \)1
Kohlrausch (strong electrolyte)\( \Lambda_m \) vs \( \sqrt{c} \)\( -K \)\( \Lambda_m^\circ \)
Nernst (298 K)\( E \) vs \( \log Q \)\( -0.0592/n \) V\( E^\circ \)
Debye–Hückel limiting law\( \log \gamma_\pm \) vs \( \sqrt{I} \)\( -A\lvert z_+ z_- \rvert \)0

Covariance

The covariance measures how two variables vary together:

$$\boxed{\text{cov}(x, y) = s_{xy} = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{n - 1} = \frac{S_{xy}}{n - 1}}$$

Each product \( (x_i - \bar{x})(y_i - \bar{y}) \) is positive when both deviations have the same sign (point in the upper-right or lower-left quadrant about the centroid) and negative otherwise.

CovariancePattern
Positive\( y \) tends to rise with \( x \)
Negative\( y \) tends to fall as \( x \) rises
Near zerono linear trend

Properties:

  • \( \text{cov}(x, x) = s_x^2 \), the variance.
  • \( \text{cov}(x, y) = \text{cov}(y, x) \).
  • \( \text{cov}(px + q,\; uy + v) = p\,u\;\text{cov}(x, y) \): shifts drop out, scale factors multiply.
  • Units: units of \( x \) × units of \( y \). Its size depends on the units chosen, so a covariance of 0.696 (ppm × absorbance) by itself says nothing about how strong the relationship is.

Covariance in error propagation

The variance of a sum or difference of two quantities that are not independent is

$$\text{var}(x \pm y) = s_x^2 + s_y^2 \pm 2\,\text{cov}(x, y)$$

The quadrature rules on the previous page assume \( \text{cov} = 0 \). The slope and intercept of a fitted line, for example, are correlated (for positive \( \bar{x} \) the covariance of \( a \) and \( b \) is negative), which is why the uncertainty of a value read off a line is not simply \( s_a \) and \( s_b \) combined in quadrature.

The Correlation Coefficient

Dividing the covariance by both SDs removes the units:

$$\boxed{r = \frac{\text{cov}(x, y)}{s_x s_y} = \frac{S_{xy}}{\sqrt{S_{xx} S_{yy}}}} \qquad S_{yy} = \sum y_i^2 - \frac{(\sum y_i)^2}{n}$$

This is Pearson’s (product-moment) correlation coefficient.

PropertyStatement
Range\( -1 \le r \le +1 \)
\( r = \pm 1 \)all points exactly on a straight line (rising or falling)
\( r = 0 \)no linear relation (there may be a curved one)
Signsame as the sign of the slope and of the covariance
Unitsnone; unchanged by change of units or origin
Under \( u = px + q \), \( v = cy + d \)\( r_{uv} = r_{xy} \) if \( p \) and \( c \) have the same sign; \( -r_{xy} \) if opposite
Symmetry\( r_{xy} = r_{yx} \)
$$\boxed{b_{yx} = r\,\frac{s_y}{s_x}} \qquad \boxed{b_{xy} = r\,\frac{s_x}{s_y}} \qquad \boxed{b_{yx}\, b_{xy} = r^2}$$

Here \( b_{yx} \) is the slope of the regression of \( y \) on \( x \) (minimising vertical misses), and \( b_{xy} \) is the slope of \( x \) on \( y \) (minimising horizontal misses).

  • The two regression lines are different unless \( r = \pm 1 \), and both pass through \( (\bar{x}, \bar{y}) \). So the means are found by solving the two line equations together.
  • \( b_{yx} \), \( b_{xy} \) and \( r \) all have the same sign.
  • Because \( r^2 \le 1 \), the product of the two slopes cannot exceed 1. This tells you which given equation is \( y \) on \( x \): try both assignments and keep the one whose slope product is at most 1.

The coefficient of determination

\( r^2 \) is the fraction of the variation in \( y \) that the straight line accounts for:

$$r^2 = 1 - \frac{\sum e_i^2}{S_{yy}}$$

\( r = 0.9 \) means \( r^2 = 0.81 \): the line explains 81% of the variance in \( y \), not 90%.

What \( r \) cannot tell you

Memory Tricks

Don’t Confuse

ThisvsThat
Regression: predicts \( y \) from \( x \) (direction matters)correlation: symmetric measure of linear association
Covariance: has units, depends on scale\( r \): dimensionless, scale-free
\( r \) = 0.9\( r^2 \) = 0.81 (fraction of variance explained)
Regression of \( y \) on \( x \): slope \( r s_y/s_x \)of \( x \) on \( y \): slope \( r s_x/s_y \) (plotted as \( y \) vs \( x \), its slope is \( 1/b_{xy} \))
\( s_{y/x} \) with \( n - 2 \)SD of replicates with \( n - 1 \)
\( r = 0 \): no linear relationindependence (stronger; \( y = x^2 \) has \( r = 0 \))
Slope of \( \ln k \) vs \( 1/T \) is \( -E_a/R \)slope of \( \log_{10} k \) vs \( 1/T \) is \( -E_a/(2.303R) \)

Common Mistakes

Practice

All questions below are original practice questions written for this page. They are not from any CSIR NET paper.

Level 1: Foundation

  1. Through which point does every least-squares line \( y = a + bx \) pass?
  2. Between what limits does the correlation coefficient lie, and what does \( r = -1 \) mean?
  3. What should be plotted to get a straight line for a second-order reaction (\( -d[\text{A}]/dt = k[\text{A}]^2 \)), and what is the slope?
Answers
  1. The centroid \( (\bar{x}, \bar{y}) \).
  2. \( -1 \le r \le 1 \). \( r = -1 \): all points lie exactly on a straight line of negative slope.
  3. \( 1/[\text{A}] \) against \( t \); slope \( k \), intercept \( 1/[\text{A}]_0 \).

Level 2: Part B Style

Practice Q1. For five data points, \( \sum x = 15 \), \( \sum y = 20 \), \( \sum x^2 = 55 \) and \( \sum xy = 66 \). The least-squares line of \( y \) on \( x \) is:

(a) \( y = 2.2 + 0.6x \) (b) \( y = 1.6 + 0.8x \) (c) \( y = 0.6 + 2.2x \) (d) \( y = 4.0 + 0.6x \)

Answer: (a)

\( S_{xy} = 66 - (15)(20)/5 = 6 \); \( S_{xx} = 55 - 15^2/5 = 10 \); \( b = 0.6 \).

\( a = \bar{y} - b\bar{x} = 4 - 0.6 \times 3 = 2.2 \).

Check: the line passes through (3, 4). (d) uses \( \bar{y} \) as the intercept and fails this check.

Practice Q2. For the points \( (-2, 4), (-1, 1), (0, 0), (1, 1), (2, 4) \), the correlation coefficient is:

(a) +1 (b) −1 (c) 0 (d) +0.5

Answer: (c) 0

\( \bar{x} = 0 \), so \( S_{xy} = \sum x_i y_i = -8 - 1 + 0 + 1 + 8 = 0 \), hence \( r = 0 \).

The data lie exactly on \( y = x^2 \). \( r = 0 \) means no linear relation, not no relation.

Practice Q3. The correlation coefficient between \( x \) and \( y \) is 0.8. If \( u = 2x + 3 \) and \( v = 5 - 3y \), the correlation coefficient between \( u \) and \( v \) is:

(a) 0.8 (b) −0.8 (c) −4.8 (d) −0.53

Answer: (b) −0.8

Shifts do not affect \( r \); the scale factors 2 and −3 have opposite signs, so the sign flips and the size is unchanged: \( r_{uv} = -0.8 \).

(c) multiplies by \( 2 \times (-3) \), which is what happens to the covariance, not to \( r \).

Practice Q4. For a reaction, a plot of \( \ln[\text{A}] \) against \( t \) (in minutes) is a straight line with slope \( -0.0300 \) min⁻¹. The half-life is about:

(a) 10.0 min (b) 16.7 min (c) 23.1 min (d) 33.3 min

Answer: (c) 23.1 min

A linear \( \ln[\text{A}] \) vs \( t \) plot means first order, with \( k = 0.0300 \) min⁻¹.

\( t_{1/2} = \ln 2/k = 0.693/0.0300 = 23.1 \) min.

(d) is \( 1/k \), the mean lifetime, not the half-life.

Level 3: Part C Style

Practice Q5. Rate constants for a reaction at four temperatures:

\( T \) / K300310330340
\( 10^3/T \) / K⁻¹3.33333.22583.03032.9412
\( \ln(k/\text{s}^{-1}) \)−3.333−2.556−1.146−0.503

Take \( R = 8.314 \) J K⁻¹ mol⁻¹. The activation energy is closest to:

(a) 26 kJ mol⁻¹ (b) 60 kJ mol⁻¹ (c) 0.060 kJ mol⁻¹ (d) 138 kJ mol⁻¹

Answer: (b) 60 kJ mol⁻¹

The points are collinear, so take the slope from the end points (a least-squares fit gives the same value):

\( \text{slope} = \frac{-0.503 - (-3.333)}{(2.9412 - 3.3333) \times 10^{-3}} = \frac{2.830}{-3.921 \times 10^{-4}} = -7.22 \times 10^{3} \) K.

\( E_a = -R \times \text{slope} = 8.314 \times 7217 = 6.00 \times 10^{4} \) J mol⁻¹ = 60.0 kJ mol⁻¹.

(a) divides by 2.303 and (d) multiplies by 2.303: both confuse natural and base-10 logs, although the data are natural logs. (c) forgets that the axis is \( 10^3/T \), so the slope read off it (−7.22) is 1000 times too small.

Practice Q6. For a set of paired data, the regression line of \( y \) on \( x \) is \( y = 0.9x + 2.4 \) and the regression line of \( x \) on \( y \) is \( x = 0.4y + 1.6 \). Consider the statements:

  1. \( \bar{x} = 4 \) and \( \bar{y} = 6 \).
  2. The correlation coefficient is +0.6.
  3. The correlation coefficient is 0.36.
  4. If \( s_x = 2 \), then \( s_y = 3 \).

Which statements are correct?

(a) 1 and 2 only (b) 1, 2 and 4 only (c) 1 and 3 only (d) 2 and 4 only

Answer: (b) 1, 2 and 4 only
  1. Both lines pass through \( (\bar{x}, \bar{y}) \). Substitute: \( x = 0.4(0.9x + 2.4) + 1.6 = 0.36x + 2.56 \Rightarrow 0.64x = 2.56 \Rightarrow \bar{x} = 4 \); \( \bar{y} = 0.9 \times 4 + 2.4 = 6 \). Correct.
  2. \( r^2 = b_{yx} b_{xy} = 0.9 \times 0.4 = 0.36 \); both slopes positive, so \( r = +0.6 \). Correct.
  3. 0.36 is \( r^2 \), not \( r \). Incorrect.
  4. \( b_{yx} = r s_y/s_x \Rightarrow s_y = 0.9 \times 2/0.6 = 3 \). Correct.

Check on the assignment: had the lines been swapped, the slopes would be \( 1/0.4 = 2.5 \) and \( 1/0.9 = 1.11 \), with product 2.78 > 1, which is impossible. So the labelling given is consistent.

Practice Q7. A fluorescence calibration with five standards gives \( n = 5 \), \( \sum x = 15 \) µM, \( \sum y = 160.5 \), \( \sum x^2 = 55 \) µM², \( \sum xy = 581.2 \) µM (\( y \) = intensity in arbitrary units). An unknown gives an intensity of 37.0. Its concentration is closest to:

(a) 3.49 µM (b) 3.71 µM (c) 3.21 µM (d) 4.03 µM

Answer: (a) 3.49 µM

\( S_{xx} = 55 - 15^2/5 = 10 \); \( S_{xy} = 581.2 - (15 \times 160.5)/5 = 581.2 - 481.5 = 99.7 \).

\( b = 99.7/10 = 9.97 \) µM⁻¹; \( a = 32.1 - 9.97 \times 3 = 2.19 \).

\( x_0 = (37.0 - 2.19)/9.97 = 3.49 \) µM.

(b) ignores the intercept: \( 37.0/9.97 = 3.71 \) µM. The unknown lies close to \( \bar{x} = 3 \) µM, the most precise region of the line.

Quick Revision Table

QuantityFormula or fact
\( S_{xx} \), \( S_{yy} \), \( S_{xy} \)\( \sum x^2 - (\sum x)^2/n \), \( \sum y^2 - (\sum y)^2/n \), \( \sum xy - \sum x \sum y/n \)
Slope, intercept\( b = S_{xy}/S_{xx} \); \( a = \bar{y} - b\bar{x} \)
Through origin\( b = \sum xy/\sum x^2 \)
Scatter about line\( s_{y/x} = \sqrt{\sum e_i^2/(n - 2)} \)
Slope, intercept SDs\( s_b = s_{y/x}/\sqrt{S_{xx}} \); \( s_a = s_{y/x}\sqrt{\sum x^2/(n S_{xx})} \)
Unknown\( x_0 = (y_0 - a)/b \)
Covariance\( S_{xy}/(n - 1) \); units of \( x \) × \( y \)
Correlation\( r = S_{xy}/\sqrt{S_{xx}S_{yy}} \), \( -1 \le r \le 1 \)
Slopes and r\( b_{yx} = r s_y/s_x \); \( b_{xy} = r s_x/s_y \); \( b_{yx} b_{xy} = r^2 \)
\( r^2 \)fraction of variance in \( y \) explained
Arrhenius plot\( \ln k \) vs \( 1/T \): slope \( -E_a/R \)
First / second order\( \ln[\text{A}] \) vs \( t \): \( -k \); \( 1/[\text{A}] \) vs \( t \): \( k \)

Key Takeaways

What’s Next

Advertisement
Advertisement