Section 7.1 Diagonalization of Symmetric Matrices

(I) Symmetric Matrices

Definition

A matrix \( A \) is symmetric if \( A^T = A \).

Facts:

It is necessarily a square matrix.
Main diagonal entries are arbitrary.
Other entries occur in pairs on opposite sides of the main diagonal.

Theorem 1

If \( A \) is symmetric, then any two eigenvectors from different eigenspaces are orthogonal.

Proof:

Let \( \vec{v}_1 \) and \( \vec{v}_2 \) be eigenvectors that correspond to distinct eigenvalues \( \lambda_1 \) and \( \lambda_2 \). We want to show \( \vec{v}_1 \cdot \vec{v}_2 = 0 \).

\[ \begin{aligned} \lambda_1(\vec{v}_1 \cdot \vec{v}_2) &= (\lambda_1\vec{v}_1)^T \vec{v}_2 = (A\vec{v}_1)^T \vec{v}_2 = \vec{v}_1^T A^T \vec{v}_2 \\ &= \vec{v}_1^T (A\vec{v}_2) = \vec{v}_1^T (\lambda_2\vec{v}_2) = \lambda_2(\vec{v}_1^T \vec{v}_2) = \lambda_2(\vec{v}_1 \cdot \vec{v}_2) \end{aligned} \]

Subtracting the right side gives:

\[ (\lambda_1 - \lambda_2)(\vec{v}_1 \cdot \vec{v}_2) = 0 \]

Since \( \lambda_1 \neq \lambda_2 \), it must be that \( \vec{v}_1 \cdot \vec{v}_2 = 0 \).

Compare:

\( A \): any square matrix \(\implies \vec{v}_1\) and \(\vec{v}_2\) are linearly independent.
\( A \): symmetric matrix \(\implies \vec{v}_1\) and \(\vec{v}_2\) are orthogonal. (Stronger)

(II) Orthogonal Diagonalization

Recall: \( A \) is diagonalizable if there exists an invertible matrix \( P \) and a diagonal matrix \( D \) such that \( A = PDP^{-1} \).

Definition

An \( n \times n \) matrix \( A \) is said to be orthogonally diagonalizable if there are an orthogonal matrix \( P \) (with \( P^{-1} = P^T \)) and a diagonal matrix \( D \) such that:

A = PDP^{-1} = PDP^T

(Recall that \( P \) is an orthogonal matrix if it is a square matrix whose column vectors are orthonormal).

Compare:

\( A \) (\( n \times n \)) \(\xrightarrow{\text{has}}\) \( n \) linearly independent eigenvectors \(\implies\) diagonalizable.
\( A \) (\( n \times n \)) \(\xrightarrow{\text{has}}\) \( n \) linearly independent orthonormal eigenvectors \(\implies\) orthogonally diagonalizable.

Theorem 2

An \( n \times n \) matrix \( A \) is orthogonally diagonalizable if and only if \( A \) is a symmetric matrix.

Proof of "\(\implies\)":

If \( A \) is orthogonally diagonalizable, then \( A = PDP^T \). Taking the transpose:

\[ A^T = (PDP^T)^T = (P^T)^T D^T P^T = P D P^T = A \]

Thus, \( A \) is symmetric.

(The "\(\impliedby\)" proof is skipped).

Example 1

Orthogonally diagonalize matrix \( A \) where \( A = \begin{bmatrix} 0 & 1 & 1 \\ 1 & 0 & 1 \\ 1 & 1 & 0 \end{bmatrix} \).

Step 1: Find all the eigenvalues.

\[ \begin{aligned} \det(A - \lambda I) &= \begin{vmatrix} -\lambda & 1 & 1 \\ 1 & -\lambda & 1 \\ 1 & 1 & -\lambda \end{vmatrix} = \begin{vmatrix} 0 & 1+\lambda & 1-\lambda^2 \\ 1 & -\lambda & 1 \\ 0 & 1+\lambda & -1-\lambda \end{vmatrix} \quad \text{(using row operations)} \\ &= -1 \begin{vmatrix} 1+\lambda & 1-\lambda^2 \\ 1+\lambda & -1-\lambda \end{vmatrix} = -(1+\lambda) \begin{vmatrix} 1 & 1-\lambda \\ 1 & -1-\lambda \end{vmatrix} \\ &= -(1+\lambda)(-1-\lambda - 1 + \lambda) = -(1+\lambda)^2(-2) \\ &= (1+\lambda)^2(2-\lambda) = 0 \end{aligned} \]

The eigenvalues are \( \lambda = -1 \) (multiplicity 2) and \( \lambda = 2 \).

Step 2: Find a basis for each eigenspace and then find an orthonormal basis using the Gram-Schmidt process.

For \( \lambda = -1 \):

\[ A - (-1)I = \begin{bmatrix} 1 & 1 & 1 \\ 1 & 1 & 1 \\ 1 & 1 & 1 \end{bmatrix} \sim \begin{bmatrix} 1 & 1 & 1 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix} \]

The dimension of the eigenspace is 2. A basis for the eigenspace of \( -1 \) is:

\[ \vec{v}_1 = \begin{bmatrix} -1 \\ 1 \\ 0 \end{bmatrix}, \quad \vec{v}_2 = \begin{bmatrix} -1 \\ 0 \\ 1 \end{bmatrix} \]

Notice that \( \vec{v}_1 \cdot \vec{v}_2 \neq 0 \), so they are not orthogonal. We apply Gram-Schmidt:

\[ \vec{z}_1 = \vec{v}_1 = \begin{bmatrix} -1 \\ 1 \\ 0 \end{bmatrix} \]

\[ \vec{z}_2 = \vec{v}_2 - \frac{\vec{v}_2 \cdot \vec{z}_1}{\vec{z}_1 \cdot \vec{z}_1}\vec{z}_1 = \begin{bmatrix} -1 \\ 0 \\ 1 \end{bmatrix} - \frac{1}{2}\begin{bmatrix} -1 \\ 1 \\ 0 \end{bmatrix} = \begin{bmatrix} -1/2 \\ -1/2 \\ 1 \end{bmatrix} \]

Normalize these vectors to form an orthonormal basis for \( \lambda = -1 \):

\[ \vec{u}_1 = \frac{\vec{z}_1}{||\vec{z}_1||} = \begin{bmatrix} -1/\sqrt{2} \\ 1/\sqrt{2} \\ 0 \end{bmatrix}, \quad \vec{u}_2 = \frac{\vec{z}_2}{||\vec{z}_2||} = \begin{bmatrix} -1/\sqrt{6} \\ -1/\sqrt{6} \\ 2/\sqrt{6} \end{bmatrix} \]

For \( \lambda = 2 \):

\[ A - 2I = \begin{bmatrix} -2 & 1 & 1 \\ 1 & -2 & 1 \\ 1 & 1 & -2 \end{bmatrix} \sim \begin{bmatrix} 1 & 0 & -1 \\ 0 & 1 & -1 \\ 0 & 0 & 0 \end{bmatrix} \implies \vec{v}_3 = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} \]

\( \vec{v}_3 \) is orthogonal to both \( \vec{v}_1 \) and \( \vec{v}_2 \) by Theorem 1. Normalizing it gives:

\[ \vec{u}_3 = \frac{\vec{v}_3}{||\vec{v}_3||} = \begin{bmatrix} 1/\sqrt{3} \\ 1/\sqrt{3} \\ 1/\sqrt{3} \end{bmatrix} \]

Step 3: Write \( A = PDP^T \).

\[ P = \begin{bmatrix} \vec{u}_1 & \vec{u}_2 & \vec{u}_3 \end{bmatrix} = \begin{bmatrix} -1/\sqrt{2} & -1/\sqrt{6} & 1/\sqrt{3} \\ 1/\sqrt{2} & -1/\sqrt{6} & 1/\sqrt{3} \\ 0 & 2/\sqrt{6} & 1/\sqrt{3} \end{bmatrix}, \quad D = \begin{bmatrix} -1 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 2 \end{bmatrix} \]

Definition

The set of eigenvalues of a matrix \( A \) is sometimes called the spectrum of \( A \).

Theorem 3 (The Spectral Theorem for Symmetric Matrices)

An \( n \times n \) symmetric matrix \( A \) has the following properties:

\( A \) has \( n \) real eigenvalues, counting multiplicities (not necessarily distinct).
The dimension of the eigenspace for each eigenvalue equals the multiplicity of \( \lambda \) as a root of the characteristic equation.
The eigenspaces are mutually orthogonal, in the sense that eigenvectors corresponding to different eigenvalues are orthogonal.
\( A \) is orthogonally diagonalizable.

Example 2 (True or False)

(1) If \( A \) is symmetric, then \( A \) always has \( n \) distinct eigenvalues.
False. They are not necessarily distinct (as seen in Example 1 where \( \lambda = -1 \) had a multiplicity of 2).

(2) If \( A \) is symmetric, then \( A^2 \) is orthogonally diagonalizable.
True. If \( A \) is symmetric, \( A^T = A \). Then \( (A^2)^T = (AA)^T = A^T A^T = A A = A^2 \), so \( A^2 \) is also symmetric. By Theorem 2, \( A^2 \) is orthogonally diagonalizable.

(3) If both \( A \) and \( B \) are orthogonally diagonalizable, then so is \( AB \).
False. If \( A \) and \( B \) are orthogonally diagonalizable, they are symmetric (\( A^T = A \) and \( B^T = B \)). For \( AB \) to be orthogonally diagonalizable, it must be symmetric. But \( (AB)^T = B^T A^T = BA \neq AB \) in general.

Counterexample:
Let \( A = \begin{bmatrix} 1 & 2 \\ 2 & 1 \end{bmatrix} \) and \( B = \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix} \). Both are symmetric.
\( AB = \begin{bmatrix} 1 & 2 \\ 2 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix} = \begin{bmatrix} 1 & -2 \\ 2 & -1 \end{bmatrix} \).
\( AB \) is not symmetric, so it is not orthogonally diagonalizable.

(III) Spectral Decomposition

Suppose \( A = PDP^T \), where the columns of \( P \) are orthonormal eigenvectors \( \vec{u}_1, \dots, \vec{u}_n \) of \( A \) and the corresponding eigenvalues \( \lambda_1, \dots, \lambda_n \) are in the diagonal matrix \( D \).

\[ \begin{aligned} A = PDP^T &= \begin{bmatrix} \vec{u}_1 & \dots & \vec{u}_n \end{bmatrix} \begin{bmatrix} \lambda_1 & \dots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \dots & \lambda_n \end{bmatrix} \begin{bmatrix} \vec{u}_1^T \\ \vdots \\ \vec{u}_n^T \end{bmatrix} \\ &= \begin{bmatrix} \lambda_1\vec{u}_1 & \dots & \lambda_n\vec{u}_n \end{bmatrix} \begin{bmatrix} \vec{u}_1^T \\ \vdots \\ \vec{u}_n^T \end{bmatrix} \end{aligned} \]

By the column-row expansion of a product (Theorem 10 in Section 2.4), we get:

A = \lambda_1\vec{u}_1\vec{u}_1^T + \lambda_2\vec{u}_2\vec{u}_2^T + \dots + \lambda_n\vec{u}_n\vec{u}_n^T

This representation of \( A \) is called a spectral decomposition of \( A \) because it breaks up \( A \) into pieces determined by the spectrum of \( A \).

Each term \( \lambda_j\vec{u}_j\vec{u}_j^T \) is an \( n \times n \) matrix of Rank 1.
Each column of \( \lambda_j\vec{u}_j\vec{u}_j^T \) is a multiple of \( \vec{u}_j \).

Example 3

Construct a spectral decomposition of \( A = \begin{bmatrix} 0 & 1 & 1 \\ 1 & 0 & 1 \\ 1 & 1 & 0 \end{bmatrix} = PDP^T \), where \( P \) and \( D \) are from Example 1.

Solution:

\[ P = \begin{bmatrix} -1/\sqrt{2} & -1/\sqrt{6} & 1/\sqrt{3} \\ 1/\sqrt{2} & -1/\sqrt{6} & 1/\sqrt{3} \\ 0 & 2/\sqrt{6} & 1/\sqrt{3} \end{bmatrix}, \quad D = \begin{bmatrix} -1 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 2 \end{bmatrix} \]

The spectral decomposition is:

\[ A = -1\vec{u}_1\vec{u}_1^T - 1\vec{u}_2\vec{u}_2^T + 2\vec{u}_3\vec{u}_3^T \]

Calculating each rank 1 matrix piece:

\[ -1\vec{u}_1\vec{u}_1^T = -1 \begin{bmatrix} -1/\sqrt{2} \\ 1/\sqrt{2} \\ 0 \end{bmatrix} \begin{bmatrix} -1/\sqrt{2} & 1/\sqrt{2} & 0 \end{bmatrix} = -\begin{bmatrix} 1/2 & -1/2 & 0 \\ -1/2 & 1/2 & 0 \\ 0 & 0 & 0 \end{bmatrix} \]

\[ -1\vec{u}_2\vec{u}_2^T = -1 \begin{bmatrix} -1/\sqrt{6} \\ -1/\sqrt{6} \\ 2/\sqrt{6} \end{bmatrix} \begin{bmatrix} -1/\sqrt{6} & -1/\sqrt{6} & 2/\sqrt{6} \end{bmatrix} = -\begin{bmatrix} 1/6 & 1/6 & -2/6 \\ 1/6 & 1/6 & -2/6 \\ -2/6 & -2/6 & 4/6 \end{bmatrix} \]

\[ 2\vec{u}_3\vec{u}_3^T = 2 \begin{bmatrix} 1/\sqrt{3} \\ 1/\sqrt{3} \\ 1/\sqrt{3} \end{bmatrix} \begin{bmatrix} 1/\sqrt{3} & 1/\sqrt{3} & 1/\sqrt{3} \end{bmatrix} = 2\begin{bmatrix} 1/3 & 1/3 & 1/3 \\ 1/3 & 1/3 & 1/3 \\ 1/3 & 1/3 & 1/3 \end{bmatrix} \]

Adding them together confirms the decomposition of \( A \).