MIT 18.701 — Lecture 9

Inner Products, Orthonormality

Recall from vector algebra that the dot product tells us a lot about the lengths of vectors and angles between vectors.

Fact (Vector Algebra). For any $\vec{x}, \vec{y} \in \mathbb{R}^n$ , with angle $\theta$ between them, we have $\vec{x} \cdot \vec{y} = |\vec{x}| \cdot |\vec{y}| \cdot \cos \theta$ .

Let's generalize this in 18.701 terms.

§ Inner Products in $\mathbb{R}^n$

Definition (Dot / Inner Product). For vectors $x, y \in \mathbb{R}^n$ , we say their dot / inner product is $\langle x, y \rangle := x^{\top}y \in \mathbb{R}$ .

Definition (Length / Norm / Distance). For a vector $x \in \mathbb{R}^n$ , the length / norm of $x$ is $|x| := \sqrt{\langle x, x \rangle}$ . Furthermore, the distance between $x, y \in \mathbb{R}^n$ is said to be $|x - y|$ .

Note that, for now, we are working only over $\mathbb{R}^n$ . We don't want to say $\left(\begin{smallmatrix} 1 \\ i \end{smallmatrix}\right)$ has length $0$ , for example—and some fields don't even have a well-defined square root!

Remark. There's also something called an outer product defined by $x \otimes y := x y^{\top} \in \mathbb{R}^{n \times n}$ .

Theorem. (Cauchy-Schwarz) For all $x, y \in \mathbb{R}^n$ , we have $|\langle x, y \rangle | \leq |x| \cdot |y|$ .

Proof: The inequality $|x + \lambda y|^2 \geq 0$ holds for all $\lambda \in \mathbb{R}$ . Expanding this, we have:

0 \leq |x + \lambda y|^2 = \langle x + \lambda y, x + \lambda y \rangle = |x|^2 + 2\langle x, y \rangle \lambda + |y|^2 \lambda^2.

The RHS of the above is a quadratic in $\lambda$ that is nonnegative everywhere, so its discriminant is nonpositive:

0 \geq \text{discriminant} = \left ( 2 \langle x, y \rangle \right ) ^2 - 4 \left ( |x|^2 \right ) \left ( |y|^2 \right ) = 4 ( \langle x, y\rangle^2 - |x|^2 |y|^2).

The above is just the statement of Cauchy-Schwarz. $\blacksquare$

Definition (Angle). The angle between $x, y \in \mathbb{R}^n$ is $\cos^{-1}\left(\frac{\langle x, y \rangle}{|x| \cdot |y|}\right)$ (which is well-defined by Cauchy-Schwarz).

§ Orthonormal Bases and Matrices

Definition (Orthogonal). Vectors $x$ and $y$ are orthogonal if $\langle x, y \rangle = 0$ , meaning the angle between them is $90^{\circ}$ .

Definition (Orthogonal / Orthonormal Bases). A basis of $\mathbb{R}^n$ is orthogonal if all vectors in the basis are pairwise orthogonal. A basis of $\mathbb{R}^n$ is orthonormal if it is orthogonal and all vectors in the basis have unit length.

Theorem. (Orthogonal $\implies$ Independent) Any set of nonzero pairwise orthogonal vectors $\{a_1, \dots, a_k \} \subseteq \mathbb{R}^n$ must be linearly independent. In particular, $n$ nonzero pairwise orthogonal vectors of $\mathbb{R}^n$ must form a basis.

Proof: Given any linear dependence of $\{a_1, \dots, a_k\}$ , we may write:

c_1a_1 + \dots + c_k a_k = 0 ~ \implies ~ \langle a_i, c_1 a_1 + \dots + c_k a_k \rangle = 0 \text{ for all } i = 1, 2, \dots, k.

But $\langle a_i, c_1 a_1 + \dots + c_k a_k \rangle = c_i |a_i|^2$ and $|a_i| \neq 0$ , so $c_i = 0$ for all $i$ . $\blacksquare$

Definition (Orthogonal Matrices). A matrix $A \in \mathbb{R}^{n \times n}$ is orthogonal if $\langle Ax, Ay \rangle = \langle x, y \rangle$ for all $x, y \in \mathbb{R}^n$ .

In other words, an orthogonal matrix is a matrix that preserves the dot product.

Theorem. (Properties of Orthogonal Matrices) For any $A \in \mathbb{R}^{n \times n}$ , the following are equivalent:

The matrix $A$ is orthogonal.
For all $x \in \mathbb{R}^n$ , we have $|Ax| = |x|$ .
We have $A^{\top}A = I_n$ .
The columns of $A$ form an orthonormal basis.
The rows of $A$ form an orthonormal basis.

Proof: We just have to show a bunch of implications.

$(\#1) \implies (\#2)$ . If $A$ is orthogonal, then $|Ax| = \sqrt{\langle Ax, Ax \rangle} = \sqrt{\langle x, x \rangle} = |x|$ .
$(\#2) \implies (\#1)$ . This follows from the polarization identity:
$\langle x, y \rangle = \dfrac{|x + y|^2 - |x|^2 - |y|^2}{2}.$
Since $A$ preserves lengths, it preserves the RHS, so it must also preserve the LHS.
$(\#3) \implies (\#1)$ . If $A^{\top}A = I_n$ , then $\langle Ax, Ay \rangle = (Ax)^{\top}(Ay) = (x^{\top}A^{\top})(Ay) = x^{\top}(A^{\top}A)y = x^{\top}y = \langle x, y \rangle$ .
$(\#1) \implies (\#3)$ . Take a pair of basis vectors $(e_i, e_j)$ , so $\langle Ae_i, Ae_j\rangle = \langle e_i, e_j \rangle$ . The LHS and RHS simplify as:
$\text{LHS} = \langle Ae_i, Ae_j \rangle = e_i^{\top}(A^{\top}A)e_j = (A^{\top}A)_{ij} ~~~~ \text{ and } ~~~~ \text{RHS} = \langle e_i, e_j \rangle = \begin{cases} 1 & i = j \\ 0 & i \neq j \end{cases}.$
Thus, the equation $\langle Ae_i, Ae_j\rangle = \langle e_i, e_j \rangle$ gives us $(A^{\top}A)_{ij} = 1$ if and only if $i = j$ , meaning $(A^{\top}A) = I_n$ .
$(\#4) \iff (\#3)$ . Suppose the columns of $A$ form vectors $\{a_1, \dots, a_n\} \subseteq \mathbb{R}^n$ . Observe that:
$A^{\top}A = \left[\begin{smallmatrix} - & a_1 & - \\ \vdots & \vdots & \vdots \\ - & a_n & - \end{smallmatrix}\right] \left[\begin{smallmatrix} \mid & \cdots & \mid \\ a_1 & \cdots & a_n \\ \mid & \cdots & \mid \end{smallmatrix}\right] = \begin{bmatrix} \langle a_i, a_j \rangle \end{bmatrix}_{i, j}.$
Thus, if $A^{\top}A = I_n$ , then $\langle a_i, a_j \rangle = 1$ if $i = j$ , and $\langle a_i, a_j \rangle = 0$ otherwise, which implies $\{a_i\}$ is an orthonormal basis. And it goes the other way, too; if $\{a_i\}$ is orthonormal, then $A^{\top}A = I_n$ .
$(\# 5) \iff (\#3)$ . If $A^{\top} A = I_n$ , then $A^{\top}$ and $A$ are inverses. So we also get $A A^{\top} = I_n$ . Now repeating the argument in $(\#3) \implies (\#4)$ with the equation $AA^{\top} = I_n$ implies the rows of $A$ are orthonormal.

And that's good enough. $\blacksquare$

§ Orthogonal Groups

Definition (Orthogonal Group). The orthogonal group is $O_n(\mathbb{R}) := \{A \in \mathbb{R}^{n \times n} \mid A^{\top} A = I_n \}$ .

Definition (Special Orthogonal Group). The special orthogonal group is $SO_n(\mathbb{R}) := \{M \in O_n(\mathbb{R}) \mid \det M = 1\}$ .

Note that for all $M \in O_n(\mathbb{R})$ , the condition $M^{\top}M = I_n$ implies $\det M = \pm 1$ . So in fact $[O_n(\mathbb{R}): SO_n(\mathbb{R})] = 2$ .

For small $n$ , what do the orthogonal groups look like?

Example. Evidently $O_1(\mathbb{R}) \cong \{ \left[\begin{smallmatrix} 1 \end{smallmatrix}\right] , \left[\begin{smallmatrix} -1 \end{smallmatrix}\right] \}$ , and $SO_1(\mathbb{R}) \cong \{\left[\begin{smallmatrix} 1 \end{smallmatrix}\right] \}$ .

Example. By inspection, for $n = 2$ , we can describe the elements of $O_2$ and $SO_2$ as rotations and reflections:

O_2(\mathbb{R}) = \left \{ \left[\begin{smallmatrix} \cos \theta & - \sin \theta \\ \sin \theta & \cos \theta \end{smallmatrix}\right], \left[\begin{smallmatrix} \cos \theta & \sin \theta \\ \sin \theta & - \cos \theta \end{smallmatrix}\right] : \theta \in [0, 2\pi) \right \} \text{ and } SO_2(\mathbb{R}) = \left \{ \left[\begin{smallmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{smallmatrix}\right]: \theta \in [0, 2\pi) \right \}.

Notice that $SO_2(\mathbb{R})$ is just the group of rotation matrices. Meanwhile, $O_2(\mathbb{R})$ includes rotations about the origin or reflections about lines through the origin.

Important! Composing reflections and rotations always yields either a pure reflection or pure rotation!

Example. For any odd $n$ and any $A \in SO_n(\mathbb{R})$ , we have:

\begin{align*}\det (A - I_n) = \ & \det (A - AA^{\top}) \\ = \ & \det (A) \det (I_n - A^{\top}) ~~~ \text{ by factoring} \\ = \ & \det (I_n - A^{\top}) ~~~ \text{ since } \det(A) = 1\\ = \ & \det ((I_n - A)^{\top}) \\ = \ & \det (I_n - A) ~~~ \text{ since } \det(M^{\top}) = \det(M) \text{ for any } M \\ = \ & (-1)^n \cdot \det(A - I_n)\end{align*}

But when $n$ is odd, the above means $\det (A - I_n) = 0$ , meaning $A$ has an eigenvector $v$ with eigenvalue $1$ . In the particular case of $n = 3$ , if we rewrite $A$ using an orthonormal basis containing $v = e_1$ , we must have:

Ae_1 = e_1 ~ \implies ~ A = \left[\begin{smallmatrix} 1 & 0 & 0 \\ 0 & \heartsuit & \heartsuit \\ 0 & \heartsuit & \heartsuit \end{smallmatrix}\right].

Note that the top entries of $A$ must be zero because the columns of $A$ must be orthonormal. And the submatrix of $\heartsuit$ s is in $SO_2(\mathbb{R})$ . So any transformation $A \in SO_3(\mathbb{R})$ may be understood as a planar rotation about an axis $v$ .

Example. It turns out that elements of $SO_4(\mathbb{R})$ (in appropriate coordinates) look like the composition of rotations in two separate two-dimensional subspaces.

§ Inner Products in Rn\mathbb{R}^nRn

§ Orthonormal Bases and Matrices

§ Orthogonal Groups

§ Inner Products in $\mathbb{R}^n$