Inner Product Spaces and Orthonormal Bases
CSIR NET linear algebra: inner products, Gram–Schmidt, projections, least squares, adjoints, orthogonal, unitary and normal matrices, spectral theorem.
- 20 min
On this page · 13 sections
Prerequisites
- Change of Basis and Canonical Forms: diagonalisation and Schur’s triangular form.
- Eigenvalues and the Cayley–Hamilton Theorem: the eigenvalue facts for special matrices are proved here.
- Metric Spaces and Several Variables: every inner product gives a norm and hence a metric.
The Hook: The Best Line Through Three Points
Inner Products
An inner product on a vector space \( V \) over \( F = \mathbb{R} \) or \( \mathbb{C} \) is a map \( \langle \cdot, \cdot \rangle : V \times V \to F \) that is
- linear in the first argument: \( \langle au + bv, w \rangle = a\langle u, w \rangle + b\langle v, w \rangle \);
- conjugate symmetric: \( \langle v, u \rangle = \overline{\langle u, v \rangle} \);
- positive definite: \( \langle v, v \rangle > 0 \) for \( v \ne 0 \).
The norm is \( \lVert v \rVert = \sqrt{\langle v, v \rangle} \).
Conditions 1 and 2 make the inner product conjugate-linear in the second argument: \( \langle u, av \rangle = \bar{a}\langle u, v \rangle \). (Physics texts put the linearity in the second slot instead; results are the same with the conjugates moved.)
| Space | Standard inner product |
|---|---|
| \( \mathbb{R}^n \) | \( \langle x, y \rangle = y^Tx = \sum x_iy_i \) |
| \( \mathbb{C}^n \) | \( \langle x, y \rangle = y^*x = \sum x_i\bar{y}_i \) |
| \( M_n(\mathbb{R}) \) | \( \langle A, B \rangle = \operatorname{tr}(B^TA) = \sum a_{ij}b_{ij} \) |
| \( C[a, b] \), real | \( \langle f, g \rangle = \int_a^b f(t)g(t)\,dt \) |
| \( \mathbb{R}^n \), weighted | \( \langle x, y \rangle = y^TMx \), \( M \) symmetric positive definite |
The last row is the general case: every inner product on \( \mathbb{R}^n \) has the form \( y^TMx \) with \( M \) symmetric positive definite (\( m_{ij} = \langle e_j, e_i \rangle \)). Testing “is this an inner product?” on \( \mathbb{R}^n \) is testing positive definiteness; see Quadratic Forms.
The basic inequalities and identities
- Triangle inequality: \( \lVert u + v \rVert \le \lVert u \rVert + \lVert v \rVert \).
- Pythagoras: if \( u \perp v \), \( \lVert u + v \rVert^2 = \lVert u \rVert^2 + \lVert v \rVert^2 \).
- Parallelogram law: \( \lVert u + v \rVert^2 + \lVert u - v \rVert^2 = 2\lVert u \rVert^2 + 2\lVert v \rVert^2 \). A norm comes from an inner product iff it satisfies this law. The norms \( \lVert x \rVert_1 \) and \( \lVert x \rVert_\infty \) on \( \mathbb{R}^2 \) fail it (try \( e_1, e_2 \) with \( \lVert \cdot \rVert_1 \): the left side is \( 4 + 4 \), the right side \( 2 + 2 \)), so they come from no inner product.
- Polarization (real case): \( \langle u, v \rangle = \frac{1}{4}\left(\lVert u + v \rVert^2 - \lVert u - v \rVert^2\right) \).
Cauchy–Schwarz is a frequent tool in Part B estimates: for example \( (x + 2y + 3z)^2 \le (1 + 4 + 9)(x^2 + y^2 + z^2) \), so the maximum of \( x + 2y + 3z \) on the unit sphere is \( \sqrt{14} \).
Orthonormal Bases and Gram–Schmidt
A set is orthogonal if its vectors are pairwise orthogonal, orthonormal if moreover each has norm 1. An orthogonal set of non-zero vectors is linearly independent.
Coordinates are free in an orthonormal basis. If \( e_1, \dots, e_n \) is orthonormal,
$$v = \sum_{i=1}^n \langle v, e_i \rangle e_i, \qquad \lVert v \rVert^2 = \sum_{i=1}^n \lvert \langle v, e_i \rangle \rvert^2 \quad \text{(Parseval)}.$$For an orthonormal set that is not a basis, \( \sum \lvert \langle v, e_i \rangle \rvert^2 \le \lVert v \rVert^2 \) (Bessel’s inequality).
From independent \( v_1, \dots, v_k \), define
$$u_1 = v_1, \qquad u_j = v_j - \sum_{i < j} \frac{\langle v_j, u_i \rangle}{\langle u_i, u_i \rangle}u_i, \qquad e_j = \frac{u_j}{\lVert u_j \rVert}.$$Then \( e_1, \dots, e_k \) is orthonormal and \( \operatorname{span}(e_1, \dots, e_j) = \operatorname{span}(v_1, \dots, v_j) \) for every \( j \). Every finite-dimensional inner product space has an orthonormal basis.
QR factorisation. Gram–Schmidt on the columns of an \( m \times n \) real matrix \( A \) of rank \( n \) gives \( A = QR \), with \( Q \) having orthonormal columns and \( R \) upper triangular with positive diagonal.
Orthogonal Complements and Projections
For a subspace \( W \), the orthogonal complement is \( W^\perp = \{ v : \langle v, w \rangle = 0 \ \forall w \in W \} \). In finite dimensions:
$$V = W \oplus W^\perp, \qquad \dim W^\perp = \dim V - \dim W, \qquad (W^\perp)^\perp = W.$$Every \( v \) splits uniquely as \( v = w + w' \) with \( w \in W \), \( w' \in W^\perp \). The orthogonal projection \( P_W(v) = w \) is the closest point of \( W \) to \( v \):
$$\lVert v - P_W v \rVert \le \lVert v - x \rVert \quad \text{for all } x \in W.$$If \( e_1, \dots, e_k \) is an orthonormal basis of \( W \), \( P_Wv = \sum \langle v, e_i \rangle e_i \). For a hyperplane \( W = \{ x \in \mathbb{R}^n : \langle x, a \rangle = 0 \} \):
$$P_Wv = v - \frac{\langle v, a \rangle}{\lVert a \rVert^2}a, \qquad \operatorname{dist}(v, W) = \frac{\lvert \langle v, a \rangle \rvert}{\lVert a \rVert}.$$Matrix of a projection. If the columns of a real \( A \) form a basis of \( W \), the projection matrix is \( P = A(A^TA)^{-1}A^T \). A real matrix \( P \) is an orthogonal projection iff \( P^2 = P = P^T \). Then its eigenvalues are 0 and 1, \( \operatorname{tr} P = \operatorname{rank} P = \dim W \), \( \lVert Px \rVert \le \lVert x \rVert \), and \( I - 2P \) fixes \( W^\perp \) and negates \( W \): it is the reflection across \( W^\perp \), and it is both symmetric and orthogonal.
Least squares
If \( Ax = b \) has no solution, a least-squares solution minimises \( \lVert Ax - b \rVert \). It makes \( Ax \) the projection of \( b \) onto the column space, i.e. the residual \( b - Ax \) is orthogonal to every column of \( A \):
$$\boxed{A^TA\,\hat{x} = A^Tb} \quad \text{(normal equations)}.$$If \( A \) has independent columns, \( A^TA \) is invertible and \( \hat{x} \) is unique.
The hook, solved. \( A = \begin{pmatrix} 1 & 0 \\ 1 & 1 \\ 1 & 2 \end{pmatrix} \), \( b = (1, 3, 4)^T \). \( A^TA = \begin{pmatrix} 3 & 3 \\ 3 & 5 \end{pmatrix} \), \( A^Tb = (8, 11)^T \). Solving: \( a = \frac{40 - 33}{6} = \frac{7}{6} \), \( b = \frac{33 - 24}{6} = \frac{3}{2} \). Residuals \( \left(-\frac{1}{6}, \frac{1}{3}, -\frac{1}{6}\right) \): they sum to 0 (orthogonal to \( (1, 1, 1) \)) and \( 0 \cdot (-\frac{1}{6}) + 1 \cdot \frac{1}{3} + 2 \cdot (-\frac{1}{6}) = 0 \) (orthogonal to \( (0, 1, 2) \)). The fitted line passes through the mean point \( (1, \frac{8}{3}) \). The same normal equations are the basis of linear regression in statistics.
Adjoints and Special Operators
Two consequences: \( \ker T^* = (\operatorname{im} T)^\perp \) and \( \operatorname{im} T^* = (\ker T)^\perp \). In matrix language, the null space of \( A^T \) is orthogonal to the column space of \( A \), which is the geometry behind the normal equations.
| Operator | Definition | Eigenvalues | Diagonalisable? |
|---|---|---|---|
| self-adjoint (Hermitian / real symmetric) | \( T^* = T \) | real | yes, orthonormal eigenbasis |
| skew-adjoint | \( T^* = -T \) | purely imaginary (or 0) | yes over \( \mathbb{C} \), unitarily |
| unitary / orthogonal | \( T^*T = I \) | \( \lvert \lambda \rvert = 1 \) | yes over \( \mathbb{C} \), unitarily |
| normal | \( T^*T = TT^* \) | any | yes, unitarily over \( \mathbb{C} \) |
| orthogonal projection | \( P^2 = P = P^* \) | 0, 1 | yes |
| positive (semi)definite | \( T = T^* \), \( \langle Tv, v \rangle > 0 \) (\( \ge 0 \)) | \( > 0 \) (\( \ge 0 \)) | yes |
Unitary and orthogonal matrices. For \( U \in M_n(\mathbb{C}) \) the following are equivalent: \( U^*U = I \); the columns are orthonormal; the rows are orthonormal; \( \lVert Ux \rVert = \lVert x \rVert \) for all \( x \); \( \langle Ux, Uy \rangle = \langle x, y \rangle \) for all \( x, y \). Real orthogonal matrices have \( \det = \pm 1 \).
- A real orthogonal \( 3 \times 3 \) matrix with \( \det = 1 \) has eigenvalue 1 (it is a rotation about an axis); with \( \det = -1 \) it has eigenvalue \( -1 \). Proof for the second: \( \det(Q + I) = \det(Q + QQ^T) = \det Q \det(I + Q^T) = -\det(Q + I) \), so \( \det(Q + I) = 0 \).
- A rotation of \( \mathbb{R}^3 \) by angle \( \theta \) has trace \( 1 + 2\cos\theta \).
- The Householder reflection \( H = I - 2uu^T \) (\( \lVert u \rVert = 1 \)) is symmetric and orthogonal, with eigenvalues \( -1 \) (once, eigenvector \( u \)) and \( 1 \) (\( n - 1 \) times), so \( \det H = -1 \).
The spectral theorem
- Complex: \( A \in M_n(\mathbb{C}) \) is unitarily diagonalisable (\( A = UDU^* \), \( U \) unitary, \( D \) diagonal) iff \( A \) is normal.
- Real: \( A \in M_n(\mathbb{R}) \) is orthogonally diagonalisable (\( A = QDQ^T \), \( Q \) orthogonal, \( D \) real diagonal) iff \( A \) is symmetric.
Why the real version needs symmetry: \( QDQ^T \) is always symmetric. Why normality is enough over \( \mathbb{C} \): by Schur \( A = UTU^* \) with \( T \) upper triangular; normality of \( A \) passes to \( T \), and a normal upper triangular matrix is diagonal (compare the \( (1, 1) \) entries of \( T^*T \) and \( TT^* \), then induct).
Further facts for a normal \( A \): \( \lVert Ax \rVert = \lVert A^*x \rVert \) for every \( x \); \( Ax = \lambda x \iff A^*x = \bar{\lambda}x \); eigenvectors for distinct eigenvalues are orthogonal.
graph TD
A["Complex square matrix A"] --> B{"A*A = AA* ?"}
B -->|no| C["Not unitarily diagonalisable
(may still be diagonalisable)"]
B -->|yes: normal| D["A = U D U*, orthonormal eigenbasis"]
D --> E{"Eigenvalues?"}
E -->|all real| F["Hermitian"]
E -->|all on unit circle| G["Unitary"]
E -->|all imaginary or 0| H["Skew-Hermitian"]
E -->|all 0 or 1| I["Orthogonal projection"]The last diagram reads both ways: a normal matrix with real eigenvalues is Hermitian, with unimodular eigenvalues unitary, and so on. Without normality the implication fails: \( \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix} \) has eigenvalue 1 only and is neither Hermitian nor unitary.
Memory Tricks
- Projection = drop the perpendicular. Subtract the component along the normal: \( v - \frac{\langle v, a \rangle}{\lVert a \rVert^2}a \).
- Parallelogram decides. A norm comes from an inner product iff the parallelogram law holds; \( \ell_1 \) and \( \ell_\infty \) fail.
- Residual is perpendicular. Least squares: \( A^T(b - A\hat{x}) = 0 \).
- Normal is the magic word. Normal \( \iff \) unitarily diagonalisable; then the eigenvalues tell you which special class it is.
- det \( -1 \) flips. A \( 3 \times 3 \) orthogonal matrix has eigenvalue \( \det Q \in \{ 1, -1 \} \).
Don’t Confuse
| This | Is not this | The difference |
|---|---|---|
| orthogonal matrix (\( Q^TQ = I \)) | matrix with orthogonal columns | columns must also have norm 1 |
| orthogonal projection (\( P^2 = P = P^T \)) | any projection (\( P^2 = P \)) | an oblique projection is not symmetric and can increase length |
| orthogonal projection matrix | orthogonal matrix | \( P \) is singular unless \( P = I \); \( Q \) is always invertible |
| \( [T^*] = [T]^* \) | in any basis | holds only in an orthonormal basis |
| diagonalisable | unitarily diagonalisable | the second needs normality; \( \begin{pmatrix} 1 & 1 \\ 0 & 2 \end{pmatrix} \) is diagonalisable, not normal |
| linear in the first slot (maths) | linear in the second slot (physics) | conventions differ; conjugates move accordingly |
| Bessel (\( \le \)) | Parseval (\( = \)) | equality needs a complete orthonormal set |
Common Mistakes
Practice
All questions below are original practice questions written for this page. They are not from any CSIR NET paper. In Level 3, as in Part C, one or more options may be correct.
Level 1: Foundation
- State the Cauchy–Schwarz inequality.
- If \( W \subseteq \mathbb{R}^5 \) has dimension 2, what is \( \dim W^\perp \)?
- Is \( \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix} \) orthogonal? Symmetric?
Answers
- \( \lvert \langle u, v \rangle \rvert \le \lVert u \rVert \lVert v \rVert \), equality iff \( u, v \) are dependent.
- 3. \( 5 - 2 \).
- Both. Its columns are \( e_2, e_1 \) (orthonormal), and it equals its transpose. It is a reflection, with eigenvalues \( 1, -1 \).
Level 2: Part B Style
Practice Q1. Applying Gram–Schmidt to \( (1, 1, 0), (1, 0, 1) \) in \( \mathbb{R}^3 \) (standard inner product), the second orthonormal vector is:
(a) \( \frac{1}{\sqrt{2}}(1, -1, 0) \) (b) \( \frac{1}{\sqrt{6}}(1, -1, 2) \) (c) \( \frac{1}{\sqrt{3}}(1, -1, 1) \) (d) \( \frac{1}{\sqrt{2}}(1, 0, 1) \)
Answer: (b)
\( u_2 = (1, 0, 1) - \frac{1}{2}(1, 1, 0) = \frac{1}{2}(1, -1, 2) \), norm \( \frac{\sqrt{6}}{2} \). Option (a) is orthogonal to \( v_1 \) but not in \( \operatorname{span}(v_1, v_2) \); (d) is \( v_2 \) normalised, not orthogonal to \( v_1 \).
Practice Q2. The orthogonal projection of \( (1, 2, 3) \) onto the plane \( x + y + z = 0 \) is:
(a) \( (2, 2, 2) \) (b) \( (-1, 0, 1) \) (c) \( (1, 0, -1) \) (d) \( (0, 1, 2) \)
Answer: (b)
\( a = (1, 1, 1) \), \( \langle v, a \rangle = 6 \), \( \lVert a \rVert^2 = 3 \): \( v - 2a = (-1, 0, 1) \). Check: the coordinates sum to 0. Option (a) is the component along the normal. The distance from the point to the plane is \( \frac{6}{\sqrt{3}} = 2\sqrt{3} \).
Practice Q3. Let \( Q \) be a real orthogonal \( 3 \times 3 \) matrix with \( \det Q = -1 \). Which number is necessarily an eigenvalue of \( Q \)?
(a) 1 (b) \( -1 \) (c) \( i \) (d) 0
Answer: (b)
\( \det(Q + I) = \det(Q + QQ^T) = \det Q \cdot \det(I + Q^T) = -\det(Q + I) \), so \( \det(Q + I) = 0 \). Option (a) can fail: \( Q = -I \). Option (d) is impossible for an invertible matrix.
Practice Q4. On \( P_1 \) (real polynomials of degree \( \le 1 \)) with \( \langle p, q \rangle = \int_0^1 p(t)q(t)\,dt \), which polynomial is orthogonal to the constant 1?
(a) \( x \) (b) \( x - 1 \) (c) \( 2x - 1 \) (d) \( x + 1 \)
Answer: (c)
\( \int_0^1 (2x - 1)\,dx = 1 - 1 = 0 \). The others integrate to \( \frac{1}{2}, -\frac{1}{2}, \frac{3}{2} \).
Level 3: Part C Style
Practice Q5. Let \( P \) be a real \( n \times n \) matrix with \( P^2 = P = P^T \) and \( \operatorname{rank} P = r \), \( 0 < r < n \). Which statements are true?
(a) Every eigenvalue of \( P \) is 0 or 1. (b) \( \operatorname{tr} P = r \). (c) \( \lVert Px \rVert \le \lVert x \rVert \) for all \( x \in \mathbb{R}^n \). (d) \( P \) is an orthogonal matrix.
Answer: (a), (b) and (c)
- (a) True. \( P^2 = P \).
- (b) True. \( P \) is diagonalisable with \( r \) ones and \( n - r \) zeros.
- (c) True. \( x = Px + (I - P)x \) and \( \langle Px, (I - P)x \rangle = x^T P^T(I - P)x = x^T(P - P^2)x = 0 \), so by Pythagoras \( \lVert x \rVert^2 = \lVert Px \rVert^2 + \lVert (I - P)x \rVert^2 \).
- (d) False. An orthogonal matrix is invertible; \( P \) has rank \( r < n \). (By contrast \( I - 2P \) is orthogonal.)
Practice Q6. The least-squares line \( y = a + bx \) is fitted to the data \( (0, 1), (1, 3), (2, 4) \). Which statements are true?
(a) \( b = \frac{3}{2} \). (b) \( a = 1 \). (c) The residual vector is orthogonal to both \( (1, 1, 1) \) and \( (0, 1, 2) \). (d) The sum of squared residuals is 0.
Answer: (a) and (c)
Normal equations: \( 3a + 3b = 8 \), \( 3a + 5b = 11 \). Subtracting, \( 2b = 3 \), \( b = \frac{3}{2} \), and \( a = \frac{8}{3} - \frac{3}{2} = \frac{7}{6} \).
- (a) True.
- (b) False. \( a = \frac{7}{6} \).
- (c) True. That is exactly what the normal equations say. Residuals \( \left(-\frac{1}{6}, \frac{1}{3}, -\frac{1}{6}\right) \) check out.
- (d) False. The sum of squares is \( \frac{1}{36} + \frac{1}{9} + \frac{1}{36} = \frac{1}{6} \). It would be 0 only if the points were collinear.
Practice Q7. Let \( A \in M_n(\mathbb{C}) \) be normal. Which statements are true?
(a) \( A \) is unitarily diagonalisable. (b) \( \lVert Ax \rVert = \lVert A^*x \rVert \) for all \( x \in \mathbb{C}^n \). (c) \( A \) is Hermitian. (d) Eigenvectors of \( A \) for distinct eigenvalues are orthogonal.
Answer: (a), (b) and (d)
- (a) True. Spectral theorem.
- (b) True. \( \lVert Ax \rVert^2 = \langle A^*Ax, x \rangle = \langle AA^*x, x \rangle = \lVert A^*x \rVert^2 \).
- (c) False. \( A = iI \) is normal (even unitary) but \( A^* = -iI \ne A \).
- (d) True. In a unitary diagonalisation the eigenspaces are spanned by disjoint sets of orthonormal columns. Directly: if \( Ax = \lambda x \), \( Ay = \mu y \), then \( A^*y = \bar{\mu}y \), and \( \lambda\langle x, y \rangle = \langle Ax, y \rangle = \langle x, A^*y \rangle = \mu\langle x, y \rangle \).
Quick Revision Table
| Idea | Fact |
|---|---|
| Inner product on \( \mathbb{R}^n \) | \( y^TMx \), \( M \) symmetric positive definite |
| Cauchy–Schwarz | \( \lvert \langle u, v \rangle \rvert \le \lVert u \rVert \lVert v \rVert \) |
| Norm from inner product | iff parallelogram law |
| ON coordinates | \( v = \sum \langle v, e_i \rangle e_i \); Parseval |
| Gram–Schmidt | \( u_j = v_j - \sum_{i < j} \frac{\langle v_j, u_i \rangle}{\lVert u_i \rVert^2}u_i \) |
| Complement | \( V = W \oplus W^\perp \), \( \dim W^\perp = n - \dim W \) |
| Projection on hyperplane | \( v - \frac{\langle v, a \rangle}{\lVert a \rVert^2}a \); distance \( \frac{\lvert \langle v, a \rangle \rvert}{\lVert a \rVert} \) |
| Projection matrix | \( P^2 = P = P^T \); \( A(A^TA)^{-1}A^T \) |
| Least squares | \( A^TA\hat{x} = A^Tb \) |
| Adjoint | conjugate transpose in an orthonormal basis; \( \ker A^* = (\operatorname{im} A)^\perp \) |
| Orthogonal \( 3 \times 3 \) | eigenvalue \( \det Q \); rotation trace \( 1 + 2\cos\theta \) |
| Spectral theorem | normal \( \iff \) unitarily diagonalisable; real symmetric \( \iff \) orthogonally diagonalisable |
Key Takeaways
What’s Next
- Quadratic Forms: the spectral theorem applied to \( x^TAx \), definiteness and Sylvester’s law of inertia.
- Change of Basis and Canonical Forms to compare unitary similarity with ordinary similarity.
- Metric Spaces and Several Variables: inner product spaces as normed and metric spaces.
- Linear Algebra Quick Revision.