Recall: \( \vec{u} \) and \( \vec{v} \) are orthogonal if \( \vec{u} \cdot \vec{v} = 0 \).

(I) Orthogonal Sets

Definition

A set of vectors \( \{\vec{u}_1, \dots, \vec{u}_p\} \) in \( \mathbb{R}^n \) is said to be an orthogonal set if each pair of distinct vectors from the set is orthogonal. That is, if:

\[ \vec{u}_i \cdot \vec{u}_j = 0 \quad \text{when } i \neq j \]

Theorem 4

If \( S = \{\vec{u}_1, \dots, \vec{u}_p\} \) is an orthogonal set of nonzero vectors in \( \mathbb{R}^n \), then \( S \) is linearly independent and hence a basis for the subspace spanned by \( S \).

(Such a set is called an orthogonal basis).

Proof of Theorem 4:

Suppose a linear combination is equal to the zero vector for some scalars \( c_1, \dots, c_p \):

\[ c_1\vec{u}_1 + \dots + c_p\vec{u}_p = \vec{0} \]

Take the dot product of both sides with \( \vec{u}_1 \):

\[ \begin{aligned} (c_1\vec{u}_1 + \dots + c_p\vec{u}_p) \cdot \vec{u}_1 &= \vec{0} \cdot \vec{u}_1 \\ c_1(\vec{u}_1 \cdot \vec{u}_1) + c_2(\vec{u}_2 \cdot \vec{u}_1) + \dots + c_p(\vec{u}_p \cdot \vec{u}_1) &= 0 \end{aligned} \]

Since \( \{\vec{u}_1, \dots, \vec{u}_p\} \) is an orthogonal set, \( \vec{u}_j \cdot \vec{u}_1 = 0 \) for \( j = 2, \dots, p \). The equation simplifies to:

\[ c_1(\vec{u}_1 \cdot \vec{u}_1) = 0 \]

Because \( \vec{u}_1 \neq \vec{0} \), we know \( \vec{u}_1 \cdot \vec{u}_1 \neq 0 \). Therefore, \( c_1 = 0 \). By applying the same process with each \( \vec{u}_j \), we find \( c_2 = 0, \dots, c_p = 0 \). Since only the trivial solution exists, the set is linearly independent.


The next theorem suggests why an orthogonal basis is nicer than other bases.

Theorem 5

Let \( \{\vec{u}_1, \dots, \vec{u}_p\} \) be an orthogonal basis for a subspace \( W \) of \( \mathbb{R}^n \). For each \( \vec{y} \) in \( W \):

\[ \vec{y} = c_1\vec{u}_1 + \dots + c_p\vec{u}_p \]

where the weights are given by:

\[ c_j = \frac{\vec{y} \cdot \vec{u}_j}{\vec{u}_j \cdot \vec{u}_j} \quad (j = 1, \dots, p) \]

Proof Idea: To find \( c_j \), compute \( \vec{y} \cdot \vec{u}_j \):

\[ \vec{y} \cdot \vec{u}_j = (c_1\vec{u}_1 + \dots + c_j\vec{u}_j + \dots + c_p\vec{u}_p) \cdot \vec{u}_j = c_j(\vec{u}_j \cdot \vec{u}_j) \]
Example 1

Express \( \vec{y} \) as a linear combination of vectors in \( S \).

\[ \vec{y} = \begin{bmatrix} 5 \\ -3 \\ 1 \end{bmatrix}, \quad S = \left\{ \begin{bmatrix} 3 \\ -3 \\ 0 \end{bmatrix}, \begin{bmatrix} 2 \\ 2 \\ -1 \end{bmatrix}, \begin{bmatrix} 1 \\ 1 \\ 4 \end{bmatrix} \right\} \]

Let's denote the vectors in \( S \) as \( \vec{v}_1, \vec{v}_2, \vec{v}_3 \).

Solution:

Old method: Solve the augmented matrix \( [\vec{v}_1 \quad \vec{v}_2 \quad \vec{v}_3 \mid \vec{y}] \).

New method: First, verify \( S \) is an orthogonal set:

\[ \begin{aligned} \vec{v}_1 \cdot \vec{v}_2 &= 3(2) + (-3)(2) + 0(-1) = 6 - 6 = 0 \\ \vec{v}_1 \cdot \vec{v}_3 &= 3(1) + (-3)(1) + 0(4) = 3 - 3 = 0 \\ \vec{v}_2 \cdot \vec{v}_3 &= 2(1) + 2(1) + (-1)(4) = 2 + 2 - 4 = 0 \end{aligned} \]

So, \( S \) is an orthogonal set and hence an orthogonal basis for \( \mathbb{R}^3 \). By Theorem 5:

\[ \begin{aligned} c_1 &= \frac{\vec{y} \cdot \vec{v}_1}{\vec{v}_1 \cdot \vec{v}_1} = \frac{5(3) + (-3)(-3) + 1(0)}{3^2 + (-3)^2 + 0^2} = \frac{15 + 9}{9 + 9} = \frac{24}{18} = \frac{4}{3} \\ c_2 &= \frac{\vec{y} \cdot \vec{v}_2}{\vec{v}_2 \cdot \vec{v}_2} = \frac{5(2) + (-3)(2) + 1(-1)}{2^2 + 2^2 + (-1)^2} = \frac{10 - 6 - 1}{4 + 4 + 1} = \frac{3}{9} = \frac{1}{3} \\ c_3 &= \frac{\vec{y} \cdot \vec{v}_3}{\vec{v}_3 \cdot \vec{v}_3} = \frac{5(1) + (-3)(1) + 1(4)}{1^2 + 1^2 + 4^2} = \frac{5 - 3 + 4}{1 + 1 + 16} = \frac{6}{18} = \frac{1}{3} \end{aligned} \]

Therefore, the decomposition is:

\[ \vec{y} = \frac{4}{3}\vec{v}_1 + \frac{1}{3}\vec{v}_2 + \frac{1}{3}\vec{v}_3 \]

(II) An Orthogonal Projection

Given a nonzero vector \( \vec{u} \) in \( \mathbb{R}^n \), consider decomposing a vector \( \vec{y} \) in \( \mathbb{R}^n \) into the sum:

\[ \vec{y} = \hat{y} + \vec{z} \]

where \( \vec{z} \) is orthogonal to \( \vec{u} \), and \( \hat{y} \) is a scalar multiple of \( \vec{u} \). Let \( \hat{y} = \alpha\vec{u} \).

Since \( \vec{z} \) is orthogonal to \( \vec{u} \):

\[ \begin{aligned} \vec{z} \cdot \vec{u} &= (\vec{y} - \hat{y}) \cdot \vec{u} = \vec{y} \cdot \vec{u} - \hat{y} \cdot \vec{u} \\ &= \vec{y} \cdot \vec{u} - \alpha(\vec{u} \cdot \vec{u}) = 0 \end{aligned} \]

Solving for \( \alpha \) gives:

\[ \alpha = \frac{\vec{y} \cdot \vec{u}}{\vec{u} \cdot \vec{u}} \]

Definition

The orthogonal projection of \( \vec{y} \) onto \( \vec{u} \) is defined as:

\[ \hat{y} = \frac{\vec{y} \cdot \vec{u}}{\vec{u} \cdot \vec{u}}\vec{u} \]

If \( L \) is the subspace (line) spanned by \( \vec{u} \), then the projection of \( \vec{y} \) onto \( L \) is denoted \( \text{Proj}_L \vec{y} \). Thus, \( \hat{y} = \text{Proj}_L \vec{y} \).

The distance from \( \vec{y} \) to \( L \) is \( ||\vec{y} - \hat{y}|| = ||\vec{z}|| \).

A geometric diagram illustrating the orthogonal projection of a vector y onto a line L. A solid gray line labeled L extends diagonally from the origin, representing the subspace spanned by a gray vector u. A blue vector labeled y originates from the origin and points away from the line. A green vector labeled z = y - y_hat drops perpendicularly from the tip of y down to the line L, forming a right angle. The resulting vector along the line L, originating from the origin to the base of z, is drawn in red and labeled y_hat = proj_L y.
Figure: Orthogonal Projection of \(\vec{y}\) onto a Line \(L\)

Note: The projection does not change if we scale the vector \( \vec{u} \) by a nonzero scalar \( c \). The projection of \( \vec{y} \) onto \( c\vec{u} \) is:

\[ \frac{\vec{y} \cdot c\vec{u}}{c\vec{u} \cdot c\vec{u}} c\vec{u} = \frac{c(\vec{y} \cdot \vec{u})}{c^2(\vec{u} \cdot \vec{u})} c\vec{u} = \frac{\vec{y} \cdot \vec{u}}{\vec{u} \cdot \vec{u}}\vec{u} \]

So, the projection is purely determined by the subspace \( L \) spanned by \( \vec{u} \).

Example 2

Let \( \vec{y} = \begin{bmatrix} -3 \\ 9 \end{bmatrix} \) and \( \vec{u} = \begin{bmatrix} 1 \\ 2 \end{bmatrix} \). Compute the distance from \( \vec{y} \) to the line through \( \vec{u} \) and the origin.

Solution: First, find the orthogonal projection \( \hat{y} \) of \( \vec{y} \) onto \( \vec{u} \):

\[ \hat{y} = \text{Proj}_L \vec{y} = \frac{\vec{y} \cdot \vec{u}}{\vec{u} \cdot \vec{u}}\vec{u} = \frac{(-3)(1) + (9)(2)}{1^2 + 2^2} \begin{bmatrix} 1 \\ 2 \end{bmatrix} = \frac{15}{5} \begin{bmatrix} 1 \\ 2 \end{bmatrix} = 3 \begin{bmatrix} 1 \\ 2 \end{bmatrix} = \begin{bmatrix} 3 \\ 6 \end{bmatrix} \]

The vector representing the distance is \( \vec{z} = \vec{y} - \hat{y} \):

\[ \vec{y} - \hat{y} = \begin{bmatrix} -3 \\ 9 \end{bmatrix} - \begin{bmatrix} 3 \\ 6 \end{bmatrix} = \begin{bmatrix} -6 \\ 3 \end{bmatrix} \]

The distance is the length of this vector:

\[ ||\vec{y} - \hat{y}|| = \left|\left| \begin{bmatrix} -6 \\ 3 \end{bmatrix} \right|\right| = \sqrt{(-6)^2 + 3^2} = \sqrt{36 + 9} = \sqrt{45} = 3\sqrt{5} \]

(III) A Geometric Interpretation of Theorem 5

Theorem 5 effectively decomposes a vector \( \vec{y} \) into a sum of orthogonal projections onto one-dimensional subspaces.

For example, if \( W = \mathbb{R}^2 = \text{Span}\{\vec{u}_1, \vec{u}_2\} \) with \( \vec{u}_1 \) and \( \vec{u}_2 \) orthogonal, we can write:

\[ \vec{y} = \hat{y}_1 + \hat{y}_2 \]

where \( \hat{y}_1 = \text{Proj}_{\vec{u}_1} \vec{y} \) and \( \hat{y}_2 = \text{Proj}_{\vec{u}_2} \vec{y} \).

(IV) Orthonormal Sets

Definition

A set \( \{\vec{u}_1, \dots, \vec{u}_p\} \) is an orthonormal set if it is an orthogonal set of unit vectors (meaning each vector has a length of \( 1 \)).

Example 3

(1) Find an orthonormal basis of the subspace spanned by the set \( S \) in Example 1.

Solution: We simply normalize each orthogonal vector from Example 1.

\[ \begin{aligned} \vec{u}_1 &= \frac{\vec{v}_1}{||\vec{v}_1||} = \frac{1}{\sqrt{18}}\begin{bmatrix} 3 \\ -3 \\ 0 \end{bmatrix} = \begin{bmatrix} 1/\sqrt{2} \\ -1/\sqrt{2} \\ 0 \end{bmatrix} \\ \vec{u}_2 &= \frac{\vec{v}_2}{||\vec{v}_2||} = \frac{1}{\sqrt{9}}\begin{bmatrix} 2 \\ 2 \\ -1 \end{bmatrix} = \begin{bmatrix} 2/3 \\ 2/3 \\ -1/3 \end{bmatrix} \\ \vec{u}_3 &= \frac{\vec{v}_3}{||\vec{v}_3||} = \frac{1}{\sqrt{18}}\begin{bmatrix} 1 \\ 1 \\ 4 \end{bmatrix} = \begin{bmatrix} 1/\sqrt{18} \\ 1/\sqrt{18} \\ 4/\sqrt{18} \end{bmatrix} \end{aligned} \]

(2) Let \( U = \begin{bmatrix} \vec{u}_1 & \vec{u}_2 & \vec{u}_3 \end{bmatrix} \). Compute \( U^T U \).

Solution:

\[ U^T U = \begin{bmatrix} \vec{u}_1^T \\ \vec{u}_2^T \\ \vec{u}_3^T \end{bmatrix} \begin{bmatrix} \vec{u}_1 & \vec{u}_2 & \vec{u}_3 \end{bmatrix} = \begin{bmatrix} \vec{u}_1 \cdot \vec{u}_1 & \vec{u}_1 \cdot \vec{u}_2 & \vec{u}_1 \cdot \vec{u}_3 \\ \vec{u}_2 \cdot \vec{u}_1 & \vec{u}_2 \cdot \vec{u}_2 & \vec{u}_2 \cdot \vec{u}_3 \\ \vec{u}_3 \cdot \vec{u}_1 & \vec{u}_3 \cdot \vec{u}_2 & \vec{u}_3 \cdot \vec{u}_3 \end{bmatrix} \]

Since the vectors form an orthonormal set, \( \vec{u}_i \cdot \vec{u}_i = 1 \) and \( \vec{u}_i \cdot \vec{u}_j = 0 \) for \( i \neq j \). Thus:

\[ U^T U = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} = I_3 \]

Theorem 6

An \( m \times n \) matrix \( U \) has orthonormal columns if and only if:

\[ U^T U = I_n \]

Theorem 7

Let \( U \) be an \( m \times n \) matrix with orthonormal columns, and let \( \vec{x} \) and \( \vec{y} \) be in \( \mathbb{R}^n \). Then:

  1. \( ||U\vec{x}|| = ||\vec{x}|| \)
    (\(\vec{x} \rightarrow U\vec{x} \) preserves lengths)
  2. \( (U\vec{x}) \cdot (U\vec{y}) = \vec{x} \cdot \vec{y} \)
  3. \( (U\vec{x}) \cdot (U\vec{y}) = 0 \) if and only if \( \vec{x} \cdot \vec{y} = 0 \)
    (\(\vec{x} \rightarrow U\vec{x} \) preserves orthogonality)

Orthogonal Matrices

If \( U \) is a square matrix with orthonormal columns, then Theorem 6 tells us that \( U^{-1} = U^T \).

Definition: An orthogonal matrix is a square invertible matrix \( U \) such that \( U^{-1} = U^T \).