Recall:

\[ \vec{y} = \hat{y} + \vec{z} \]

where \( \vec{z} = \vec{y} - \hat{y} \) is orthogonal to \( \vec{u} \), and \( \hat{y} = \text{Proj}_L \vec{y} \).

We now replace the line \( L \) with a \( p \)-dimensional vector space \( W \).

Outline:

  1. \( \vec{y} - \hat{y} \) is orthogonal to \( W \) and \( ||\vec{y} - \hat{y}|| \) is the distance from \( \vec{y} \) to \( W \).
  2. \( \hat{y} \) is the unique vector in \( W \) for which \( \vec{y} - \hat{y} \) is orthogonal to \( W \).
  3. \( \hat{y} \) is the unique vector in \( W \) that is closest to \( \vec{y} \).

(I) Orthogonal Projections

Let \( \{\vec{u}_1, \dots, \vec{u}_n\} \) be a basis for \( \mathbb{R}^n \).

Write \( \vec{y} \) as:

\[ \vec{y} = \vec{z}_1 + \vec{z}_2 \]

where \( \vec{z}_1 \) is a linear combination of some \( \vec{u}_i \)'s, and \( \vec{z}_2 \) is a linear combination of the other \( \vec{u}_i \)'s.

If \( \{\vec{u}_1, \dots, \vec{u}_n\} \) is an orthogonal basis, then \( \vec{z}_2 \in W^\perp \) if \( \vec{z}_1 \in W \).

Theorem 8 (The Orthogonal Decomposition Theorem)

Let \( W \) be a subspace of \( \mathbb{R}^n \) and let \( \{\vec{u}_1, \dots, \vec{u}_p\} \) be an orthogonal basis of \( W \). Then each \( \vec{y} \) in \( \mathbb{R}^n \) can be written uniquely in the form:

\[ \vec{y} = \hat{y} + \vec{z} \]

where \( \hat{y} \in W \) and \( \vec{z} \in W^\perp \).

In fact, \( \hat{y} \) is given by:

\[ \hat{y} = \text{Proj}_W \vec{y} = \frac{\vec{y} \cdot \vec{u}_1}{\vec{u}_1 \cdot \vec{u}_1}\vec{u}_1 + \dots + \frac{\vec{y} \cdot \vec{u}_p}{\vec{u}_p \cdot \vec{u}_p}\vec{u}_p \quad \text{(1)} \]

Proof:

"Existence": Similar to Theorem 5.

"Uniqueness": Assume \( \vec{y} = \hat{y} + \vec{z} = \hat{y}_1 + \vec{z}_1 \). Then:

\[ \underbrace{\hat{y} - \hat{y}_1}_{\in W} = \underbrace{\vec{z}_1 - \vec{z}}_{\in W^\perp} \]

Since the left side is in \( W \) and the right side is in \( W^\perp \), and \( W \cap W^\perp = \{\vec{0}\} \), they must both equal the zero vector. So, \( \hat{y} - \hat{y}_1 = \vec{0} \) and \( \vec{z}_1 - \vec{z} = \vec{0} \), meaning \( \hat{y} = \hat{y}_1 \) and \( \vec{z} = \vec{z}_1 \).

The uniqueness of the decomposition shows that the orthogonal projection \( \hat{y} \) depends only on \( W \) and not on the particular basis of \( W \).

Equation (1) shows that \( \text{Proj}_W \vec{y} \) is the sum of its projections onto one-dimensional subspaces that are mutually orthogonal.

Example 1

Let \( \vec{u}_1 = \begin{bmatrix} 2 \\ 5 \\ -1 \end{bmatrix} \), \( \vec{u}_2 = \begin{bmatrix} -2 \\ 1 \\ -1 \end{bmatrix} \), and \( \vec{y} = \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} \). Let \( W = \text{Span}\{\vec{u}_1, \vec{u}_2\} \).

(1) Write \( \vec{y} \) as the sum of a vector in \( W \) and a vector in \( W^\perp \).

Solution: Observe that \( \vec{u}_1 \cdot \vec{u}_2 = 2(-2) + 5(1) + (-1)(-1) = 0 \), and hence \( \{\vec{u}_1, \vec{u}_2\} \) is an orthogonal basis of \( W \).

\[ \begin{aligned} \hat{y} &= \frac{\vec{y} \cdot \vec{u}_1}{\vec{u}_1 \cdot \vec{u}_1}\vec{u}_1 + \frac{\vec{y} \cdot \vec{u}_2}{\vec{u}_2 \cdot \vec{u}_2}\vec{u}_2 \\ &= \frac{9}{30} \begin{bmatrix} 2 \\ 5 \\ -1 \end{bmatrix} + \frac{-3}{6} \begin{bmatrix} -2 \\ 1 \\ -1 \end{bmatrix} \\ &= \begin{bmatrix} 3/5 \\ 3/2 \\ -3/10 \end{bmatrix} + \begin{bmatrix} 1 \\ -1/2 \\ 1/2 \end{bmatrix} \dots \quad \text{(Simplifying directly gives:)} \\ &= \begin{bmatrix} -2/5 \\ 2 \\ 1/5 \end{bmatrix} \in W \end{aligned} \]
\[ \vec{z} = \vec{y} - \hat{y} = \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} - \begin{bmatrix} -2/5 \\ 2 \\ 1/5 \end{bmatrix} = \begin{bmatrix} 7/5 \\ 0 \\ 14/5 \end{bmatrix} \in W^\perp \]

Check: \( \hat{y} \cdot \vec{z} = (-2/5)(7/5) + (2)(0) + (1/5)(14/5) = -14/25 + 0 + 14/25 = 0 \).

(II) The Best Approximation Theorem

Theorem 9 (The Best Approximation Theorem)

Let \( W \) be a subspace of \( \mathbb{R}^n \). Let \( \vec{y} \) be any vector in \( \mathbb{R}^n \) and let \( \hat{y} = \text{Proj}_W \vec{y} \).

Then \( \hat{y} \) is the closest point in \( W \) to \( \vec{y} \) in the sense that:

\[ ||\vec{y} - \hat{y}|| < ||\vec{y} - \vec{v}|| \]

for all \( \vec{v} \) in \( W \) distinct from \( \hat{y} \).

A 3D geometric diagram proving the Best Approximation Theorem. A light gray plane represents a 2D subspace W. A blue vector y extends upwards, above the plane. A red vector y_hat lies flat on the plane W, representing the orthogonal projection of y onto W. A green dashed line drops straight down from the tip of y to y_hat, labeled y - y_hat, forming a 3D right angle with the plane. Another arbitrary gray vector v also lies on the plane W. A dashed black line labeled y - v connects the tip of y to v, forming the hypotenuse of a right triangle suspended in 3D space. The leg on the plane is labeled y_hat - v. This right triangle visually demonstrates that the perpendicular drop ||y - y_hat|| is the shortest possible distance to the plane.
Figure: The Best Approximation Theorem (3D Right Triangle Proof)

Facts:

  1. \( \vec{y} - \hat{y} \) is in \( W^\perp \implies \vec{y} - \hat{y} \) is perpendicular to \( W \).
  2. \( ||\vec{y} - \hat{y}|| < ||\vec{y} - \vec{v}|| \) represents the right triangle hypotenuse relationship.
  3. \( ||\vec{y} - \hat{y}|| \) is the distance from \( \vec{y} \) to \( W \).
  4. \( \hat{y} \) is the unique closest point in \( W \) to \( \vec{y} \).
  5. \( \hat{y} \) is the best approximation to \( \vec{y} \) by the elements of \( W \).
  6. \( ||\vec{y} - \vec{v}|| \) is the "error" of using \( \vec{v} \) in place of \( \vec{y} \).
Example 1 (Continued)

(2) Find the closest point in \( W \) to \( \vec{y} \).

Solution: The closest point is the projection \( \hat{y} \):

\[ \hat{y} = \begin{bmatrix} -2/5 \\ 2 \\ 1/5 \end{bmatrix} \]

(3) Find the distance from \( \vec{y} \) to \( W \).

Solution: The distance is the length of \( \vec{z} \):

\[ ||\vec{y} - \hat{y}|| = ||\vec{z}|| = \left|\left| \begin{bmatrix} 7/5 \\ 0 \\ 14/5 \end{bmatrix} \right|\right| = \frac{7}{5}\sqrt{1^2 + 0^2 + 2^2} = \frac{7}{5}\sqrt{5} \]

(III) Projections with Orthonormal Bases

The final theorem in this section shows how the formula in Theorem 8 is simplified when the basis for \( W \) is an orthonormal set (meaning \( \vec{u}_i \cdot \vec{u}_i = 1 \) for all \( i \)).

Theorem 10

If \( \{\vec{u}_1, \dots, \vec{u}_p\} \) is an orthonormal basis for a subspace \( W \) of \( \mathbb{R}^n \), then:

\[ \text{Proj}_W \vec{y} = (\vec{y} \cdot \vec{u}_1)\vec{u}_1 + \dots + (\vec{y} \cdot \vec{u}_p)\vec{u}_p \]

If \( U = \begin{bmatrix} \vec{u}_1 & \dots & \vec{u}_p \end{bmatrix} \), then:

\[ \text{Proj}_W \vec{y} = U U^T \vec{y} \]

for all \( \vec{y} \) in \( \mathbb{R}^n \).

Sketch of Proof:

\( \vec{y} \) is a linear combination of \( \vec{u}_1, \dots, \vec{u}_p \) with weights \( \vec{y} \cdot \vec{u}_1, \dots, \vec{y} \cdot \vec{u}_p \).

\[ \text{Proj}_W \vec{y} = \begin{bmatrix} \vec{u}_1 & \dots & \vec{u}_p \end{bmatrix} \begin{bmatrix} \vec{u}_1^T \vec{y} \\ \vdots \\ \vec{u}_p^T \vec{y} \end{bmatrix} = U (U^T \vec{y}) = U U^T \vec{y} \]
Example 2

Let \( \vec{v}_1 = \begin{bmatrix} 2 \\ 1 \\ 2 \end{bmatrix} \), \( \vec{v}_2 = \begin{bmatrix} -2 \\ 2 \\ 1 \end{bmatrix} \), and \( \vec{v}_3 = \begin{bmatrix} 4 \\ 8 \\ 1 \end{bmatrix} \).

Note that \( \vec{v}_1 \) and \( \vec{v}_2 \) are orthogonal, but \( \vec{v}_3 \) is not orthogonal to \( \vec{v}_1 \) and \( \vec{v}_2 \). It can be shown that \( \vec{v}_3 \) is not in the subspace \( W \) spanned by \( \vec{v}_1 \) and \( \vec{v}_2 \). Use this fact to find a nonzero \( \vec{z} \) in \( \mathbb{R}^3 \) that is orthogonal to \( \vec{v}_1 \) and \( \vec{v}_2 \).

Solution: Apply Theorem 8 with \( \vec{y} = \vec{v}_3 \) and \( W = \text{Span}\{\vec{v}_1, \vec{v}_2\} \).

\[ \begin{aligned} \hat{v}_3 &= \frac{\vec{v}_3 \cdot \vec{v}_1}{\vec{v}_1 \cdot \vec{v}_1}\vec{v}_1 + \frac{\vec{v}_3 \cdot \vec{v}_2}{\vec{v}_2 \cdot \vec{v}_2}\vec{v}_2 \\ &= \begin{bmatrix} 2 \\ 4 \\ 5 \end{bmatrix} \end{aligned} \]
\[ \vec{z} = \vec{v}_3 - \hat{v}_3 = \begin{bmatrix} 4 \\ 8 \\ 1 \end{bmatrix} - \begin{bmatrix} 2 \\ 4 \\ 5 \end{bmatrix} = \begin{bmatrix} 2 \\ 4 \\ -4 \end{bmatrix} \in W^\perp \]
Example 3

Find an orthonormal basis \( \{\vec{u}_1, \vec{u}_2\} \) of \( W \) from Example 2 and compute \( U^T U \) and \( U U^T \) for \( U = \begin{bmatrix} \vec{u}_1 & \vec{u}_2 \end{bmatrix} \).

Solution: Normalize \( \vec{v}_1 \) and \( \vec{v}_2 \):

\[ \vec{u}_1 = \frac{1}{3}\begin{bmatrix} 2 \\ 1 \\ 2 \end{bmatrix}, \quad \vec{u}_2 = \frac{1}{3}\begin{bmatrix} -2 \\ 2 \\ 1 \end{bmatrix} \]
\[ U = \begin{bmatrix} 2/3 & -2/3 \\ 1/3 & 2/3 \\ 2/3 & 1/3 \end{bmatrix} \]

Compute \( U^T U \):

\[ U^T U = \begin{bmatrix} 2/3 & 1/3 & 2/3 \\ -2/3 & 2/3 & 1/3 \end{bmatrix} \begin{bmatrix} 2/3 & -2/3 \\ 1/3 & 2/3 \\ 2/3 & 1/3 \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \]

(Matches Theorem 6 in Section 6.2)

Compute \( U U^T \):

\[ U U^T = \begin{bmatrix} 2/3 & -2/3 \\ 1/3 & 2/3 \\ 2/3 & 1/3 \end{bmatrix} \begin{bmatrix} 2/3 & 1/3 & 2/3 \\ -2/3 & 2/3 & 1/3 \end{bmatrix} = \begin{bmatrix} 8/9 & -2/9 & 2/9 \\ -2/9 & 5/9 & 4/9 \\ 2/9 & 4/9 & 5/9 \end{bmatrix} \]

Check: We can verify that \( \text{Proj}_W \vec{v}_3 \) (which is \( \hat{v}_3 \) in Example 2) equals \( (U U^T)\vec{v}_3 \).