Understanding Singular Value Decomposition

We are going to compare and contrast the diagonalization of a matrix (eigenspace decomposition) with singular value decomposition. One could always stick with the strict definitions, and say that the latter is the same thing, only for a different matrix. This is true, but I would like to take the route of watching diagonalizability and eigencoordinates break down, piece by piece, and then seeing how SVD can help in each of these situations.

Setup: Any time we wish to talk about singular value decomposition, there is always an inner product involved. We will use the (real) dot product for these examples.

Example 1: Diagonal with Positive Eigenvalues

A = \left[ \begin{matrix} 1.1 & 0 \\ 0 & 1.5 \end{matrix} \right]

A^T A = \left[ \begin{matrix} 1.21 & 0 \\ 0 & 2.25 \end{matrix} \right] \

The eigenaxes of $A$ are the $x$ -axis and $y$ -axis.
$A$ 's eigenvalues are $1.1$ and $1.5,$ the same as its singular values
Observation: $A^T = A,$ so $A^T A = A^2.$ No differences yet.

Example 2: Diagonalizable with Orthogonal Eigenaxes

A = \frac{1}{25} \left[ \begin{matrix} 41 & 12 \\ 12 & 34 \end{matrix} \right]

A^T A = \frac{1}{25} \left[ \begin{matrix} 73 & 36 \\ 36 & 52 \end{matrix} \right] \ \ \ \ \

The eigenaxes of $A$ are orthogonal to each other.
$A$ 's eigenvalues are $2$ and $1,$ the same as its singular values
Observation: $A^T = A,$ so $A^T A = A^2.$ Still no differences.

Example 3: Diagonalizable with Orthogonal Eigenaxes, but Not Positive

Now we have the famous Fibonacci matrix that sends two adjacent Fibonacci numbers to the next pair in the sequence: $(F_{n-1}, F_n) \mapsto (F_n, F_{n+1}).$

A = \left[ \begin{matrix} 0 & 1 \\ 1 & 1 \end{matrix} \right]

A^T A = \left[ \begin{matrix} 1 & 1 \\ 1 & 2 \end{matrix} \right] \ \ \ \ \

$A$ 's eigenvalues are the golden ratio $\varphi$ and its conjugate $\overline{\varphi}.$
The eigenaxes of $A$ are orthogonal to each other.
Observation: $A^T = A,$ so it's still true that $A^T A = A^2.$

But despite $A^T A = A^2,$ we have our first difference. $\varphi \approx 1.62$ and $\overline{\varphi} \approx -0.62,$ a negative eigenvalue. Singular values are always positive, but since the eigenaxes are perpendicular, this just means that the eigenaxis corresponding to $\overline{\varphi}$ does not reflect, even though it scales down by the same amount.

Why should the direction switch to allow for a positive multiplier? Let's add a unit circle to the domain and look at its image to get an idea of why. What do you see when you look at the image of the unit circle?

Example 4: Diagonalizable without Orthogonal Eigenaxes

A = \left[ \begin{matrix} 5 & -2 \\ 2 & 0 \end{matrix} \right]

A^T A = \left[ \begin{matrix} 29 & 10 \\ 10 & 4 \end{matrix} \right] \ \ \ \ \

$A$ 's eigenvalues are $4$ and $1.$
The eigenaxes of $A$ are nowhere near orthogonal.
Observation: $A^T$ and $A$ are no longer equal.

The singular values of $A$ are approximately $5.7$ and $0.7,$ quite the contrast from the eigenvalues of $A.$

Example 5: Only One Eigenaxis

A = \left[ \begin{matrix} 1 & 1 \\ 0 & 1 \end{matrix} \right]

A^T A = \left[ \begin{matrix} 2 & 1 \\ 1 & 1 \end{matrix} \right] \ \ \ \ \

$A$ 's only eigenvalue is $1.$
There is only one eigenaxis.
Observation: $A^T A$ and $A$ are no longer equal, but $A^T A$ is the same as that of the Fibonacci matrix.

So its SVD axes in the domain are the same, and so are its singular values. The outputs are not the same. The dimensions of the ellipse are the same, but it is in a different position. Can you find the square root of $A^T A?$