(I) The Inner Product

If $\vec{u}=\begin{bmatrix}u_1\\\vdots\\u_n\end{bmatrix}$ and $\vec{v}=\begin{bmatrix}v_1\\\vdots\\v_n\end{bmatrix}$, then the inner product of $\vec{u}$ and $\vec{v}$ is:

$$\vec{u}\cdot\vec{v}=\vec{u}^T\vec{v}=\begin{bmatrix}u_1&\dots&u_n\end{bmatrix}\begin{bmatrix}v_1\\\vdots\\v_n\end{bmatrix}=u_1v_1+\dots+u_nv_n\in\mathbb{R}$$
Example 1

Compute $\vec{u}\cdot\vec{v}$, $\vec{v}\cdot\vec{u}$ and $\vec{u}\cdot\vec{u}$ for $\vec{u}=\begin{bmatrix}2\\-5\\-1\end{bmatrix}$ and $\vec{v}=\begin{bmatrix}3\\2\\-3\end{bmatrix}$.

Solution:

$$\begin{aligned}\vec{u}\cdot\vec{v}&=2\cdot3+(-5)\cdot2+(-1)(-3)=-1\\\vec{v}\cdot\vec{u}&=3\cdot2+2\cdot(-5)+(-3)(-1)=-1\\\vec{u}\cdot\vec{u}&=2^2+(-5)^2+(-1)^2=30\end{aligned}$$

Theorem 1

Let $\vec{u}$, $\vec{v}$ and $\vec{w}$ be vectors in $\mathbb{R}^n$ and $c$ be a scalar.

  1. $\vec{u}\cdot\vec{v}=\vec{v}\cdot\vec{u}$ (commutative)
  2. $(\vec{u}+\vec{v})\cdot\vec{w}=\vec{u}\cdot\vec{w}+\vec{v}\cdot\vec{w}$ (distributive)
  3. $(c\vec{u})\cdot\vec{v}=c(\vec{u}\cdot\vec{v})=\vec{u}\cdot(c\vec{v})$
  4. $\vec{u}\cdot\vec{u}\ge0$ and $\vec{u}\cdot\vec{u}=0$ if and only if $\vec{u}=\vec{0}$

Note: $\vec{u}\cdot\vec{u}=u_1^2+u_2^2+\dots+u_n^2\ge0$. Also, $\vec{u}\cdot\vec{u}=0$ if and only if $u_i=0$ for all $1\le i\le n$.

Combining (b) and (c), we have:

$$(c_1\vec{u}_1+\dots+c_p\vec{u}_p)\cdot\vec{w}=c_1(\vec{u}_1\cdot\vec{w})+\dots+c_p(\vec{u}_p\cdot\vec{w})$$

(II) The Length of a Vector

Definition

The length or norm of $\vec{v}$ is:

$$||\vec{v}||=\sqrt{\vec{v}\cdot\vec{v}}=\sqrt{v_1^2+\dots+v_n^2}$$

(It is well-defined since $\vec{v}\cdot\vec{v}\ge0$).

For example, if $\vec{v}=\begin{bmatrix}a\\b\end{bmatrix}\in\mathbb{R}^2$, then $||\vec{v}||=\sqrt{a^2+b^2}$ by the Pythagorean Theorem.

Property:

\[||c\vec{v}||=|c|||\vec{v}||\]

Example 2

Find the length of $\vec{v}=\begin{bmatrix}1/3\\-1/3\\\sqrt{2}/6\end{bmatrix}$.

Solution:

$$||\vec{v}||=\left|\left|\frac{1}{6}\begin{bmatrix}2\\-2\\\sqrt{2}\end{bmatrix}\right|\right|=\frac{1}{6}\sqrt{2^2+(-2)^2+(\sqrt{2})^2}=\frac{1}{6}\sqrt{10}$$

Definition

A unit vector is a vector whose length is $1$.

Normalizing: $\vec{u}=\frac{\vec{v}}{||\vec{v}||}$ forms a unit vector in the same direction as $\vec{v}$.

Example 3

Find a unit vector $\vec{u}$ in the same direction as $\vec{v}$ in Example 2.

Solution:

$$\begin{aligned}\vec{u}&=\frac{\vec{v}}{||\vec{v}||}=\frac{6}{\sqrt{10}}\begin{bmatrix}1/3\\-1/3\\\sqrt{2}/6\end{bmatrix}=\begin{bmatrix}2/\sqrt{10}\\-2/\sqrt{10}\\\sqrt{2}/\sqrt{10}\end{bmatrix}\\&=\begin{bmatrix}2/\sqrt{10}\\-2/\sqrt{10}\\1/\sqrt{5}\end{bmatrix}\end{aligned}$$

We may also normalize the scaled vector $\begin{bmatrix}2\\-2\\\sqrt{2}\end{bmatrix}$ directly:

$$\vec{u}=\frac{1}{\sqrt{10}}\begin{bmatrix}2\\-2\\\sqrt{2}\end{bmatrix}=\begin{bmatrix}2/\sqrt{10}\\-2/\sqrt{10}\\1/\sqrt{5}\end{bmatrix}$$

(III) Distance in $\mathbb{R}^n$

Definition

For $\vec{u}$ and $\vec{v}$ in $\mathbb{R}^n$, the distance between $\vec{u}$ and $\vec{v}$ is:

$$\text{dist}(\vec{u},\vec{v})=||\vec{u}-\vec{v}||=\sqrt{(u_1-v_1)^2+\dots+(u_n-v_n)^2}$$

(IV) Orthogonal Vectors

By the Pythagorean Theorem, if two vectors $u$ and$v$ are orthogonal, then $\text{dist}(\vec{u},\vec{v})=\text{dist}(\vec{u},-\vec{v})=||\vec{u}+\vec{v}||$ when they are orthogonal. Expanding the squared length yields:

$$\begin{aligned}||\vec{u}+\vec{v}||^2&=(\vec{u}+\vec{v})\cdot(\vec{u}+\vec{v})\\&=\vec{u}\cdot\vec{u}+\vec{u}\cdot\vec{v}+\vec{v}\cdot\vec{u}+\vec{v}\cdot\vec{v}\\&=||\vec{u}||^2+2(\vec{u}\cdot\vec{v})+||\vec{v}||^2\end{aligned}$$

If $\vec{u}\cdot\vec{v}=0$, then $||\vec{u}+\vec{v}||^2=||\vec{u}||^2+||\vec{v}||^2$.

Definition

Two vectors $\vec{u}$ and $\vec{v}$ in $\mathbb{R}^n$ are orthogonal to each other if $\vec{u}\cdot\vec{v}=0$.

Theorem 2 (Pythagorean Theorem)

Two vectors $\vec{u}$ and $\vec{v}$ are orthogonal if and only if:

$$||\vec{u}+\vec{v}||^2=||\vec{u}||^2+||\vec{v}||^2$$

HW #32: Prove $||\vec{u}+\vec{v}||^2+||\vec{u}-\vec{v}||^2=2||\vec{u}||^2+2||\vec{v}||^2$.

(V) Orthogonal Complements

Definitions

If a vector $\vec{z}$ is orthogonal to every vector in a subspace $W$ of $\mathbb{R}^n$, then $\vec{z}$ is orthogonal to $W$.

The set of all vectors $\vec{z}$ that are orthogonal to $W$ is called the orthogonal complement of $W$, denoted by $W^\perp$.

Facts: Let $W$ be a subspace of $\mathbb{R}^n$.

  1. $\vec{x}$ is in $W^\perp$ if and only if $\vec{x}$ is orthogonal to every vector in a set that spans $W$. (#36, 37)
  2. $W^\perp$ is a subspace of $\mathbb{R}^n$. (#38)
  3. $W\cap W^\perp=\{\vec{0}\}$. (#39) (Because $\vec{u}\cdot\vec{u}=0$ if and only if $\vec{u}=\vec{0}$).
  4. $\dim W+\dim W^\perp=n$. (Practice problem #3)
Example 4

Let $W$ be a plane through the origin in $\mathbb{R}^3$ and let $L$ be the line through the origin and perpendicular to $W$.

Then $W=L^\perp$ and $L=W^\perp$. For any $\vec{w}$ in $W$ and $\vec{z}$ in $L$, $\vec{w}\cdot\vec{z}=0$.

Note: A common way to prove two sets $S=T$ is to show that $S$ is a subset of $T$ and $T$ is a subset of $S$.

Theorem 3

Let $A$ be an $m\times n$ matrix. Then:

$$(\text{Row }A)^\perp=\text{Nul }A\quad\text{and}\quad(\text{Col }A)^\perp=\text{Nul }A^T$$

Proof of Theorem 3:

Let $A=\begin{bmatrix}\vec{r}_1\\\vdots\\\vec{r}_m\end{bmatrix}$ and $\vec{x}=\begin{bmatrix}x_1\\\vdots\\x_n\end{bmatrix}$. Then $A\vec{x}=\begin{bmatrix}\vec{r}_1\cdot\vec{x}\\\vec{r}_2\cdot\vec{x}\\\vdots\\\vec{r}_m\cdot\vec{x}\end{bmatrix}$.

Part 1: $(\text{Row }A)^\perp\subseteq\text{Nul }A$
If $\vec{x}$ is orthogonal to $\text{Row }A$, then $\vec{x}$ is orthogonal to each row of $A$. Therefore, $A\vec{x}=\vec{0}$ and $\vec{x}\in\text{Nul }A$.

Part 2: $\text{Nul }A\subseteq(\text{Row }A)^\perp$
If $\vec{x}\in\text{Nul }A$, then $A\vec{x}=\vec{0}$, meaning $\vec{x}$ is orthogonal to each row of $A$. Since the rows of $A$ span $\text{Row }A$, we have $\vec{x}\in(\text{Row }A)^\perp$.

Written HW #34

Let $\vec{u}=\begin{bmatrix}5\\-6\\7\end{bmatrix}$ and let $W$ be the set of all $\vec{x}$ in $\mathbb{R}^3$ such that $\vec{u}\cdot\vec{x}=0$. Show that $W$ is a subspace of $\mathbb{R}^3$. Describe $W$ in geometric language.

Hint: Consider the homogeneous system $\vec{u}^T\vec{x}=0$.
$W=\text{Nul}\begin{bmatrix}5&-6&7\end{bmatrix}$.

Example 5

Let $W$ be a subspace of $\mathbb{R}^n$. Prove $\dim W+\dim W^\perp=n$.

Solution:

Consider $W\neq\{\vec{0}\}$. Let $\vec{b}_1,\dots,\vec{b}_p$ be a basis of $W$, where $1\le p\le n$.

Let $A$ be the $p\times n$ matrix having rows $\vec{b}_1^T,\dots,\vec{b}_p^T$.

It follows that $W=\text{Row }A$. Theorem 3 implies that:

$$W^\perp=(\text{Row }A)^\perp=\text{Nul }A$$

Therefore, $\dim W^\perp=\dim\text{Nul }A$. By the Rank Theorem, the number of pivot columns plus the number of free variables equals $n$, meaning $\dim\text{Row }A+\dim\text{Nul }A=n$.

Since $\dim\text{Row }A=\dim W$, we conclude:

$$\dim W+\dim W^\perp=n$$