7E Singular Value Decomposition

Suppose $T \in \mathscr{L}(V,W) .$ Then a) $T^* T$ is a positive operator on $V.$ b) $\ker{\left(T^* T\right)}=\ker⁡(T)$ c) $T^* T\left(V\right)=T^* (V)$ d) $\dim{\left(T\left(V\right)\right)}=\dim{\left(T^* \left(V\right)\right)}=\dim{\left(T^* T\left(V\right)\right)}$

a) We first verify that $T^* T$ is self-adjoint: ${\left(T^* T\right)}^* =T^* {\left(T^* \right)}^* =T^* T .$ To conclude, we let $v∈V$ and show:

b) We will show the first inclusion, since the second is clear: If $\mathrm{v}\in \ker{({\mathrm{T}}^* \mathrm{T})},$ then

so $\mathrm{T}\left(\mathrm{v}\right)=0.$ c) This part follows from part (b) and the kernel/range relationship between adjoints:

d) The first equality follows from the setup in 3F, where the vectors $\left({\mathrm{w}}_{1},\ldots ,\ {\mathrm{w}}_{\mathrm{\ell }}\right)$ were a basis for $T(V)$ and the corresponding dual basis vectors $\left({\mathrm{\omega }}_{1},\ \ldots ,{\mathrm{\omega }}_{\mathrm{\ell }}\right)$ in $W$ were a basis for $\ker⁡({\mathrm{T}}^{\prime }) .$ Now, $\ker⁡({\mathrm{T}}^{\prime })$ and $\ker⁡({\mathrm{T}}^* )$ have the same dimension, and in fact if we had constructed our setup for 3F with orthonormal bases and extension (in particular, if Gram-Schmidtt had been applied to the ${\mathrm{w}}_{\mathrm{i}}$'s before extending to a basis for $W$) then they would have been corresponding vectors, i.e., $\left({\mathrm{w}}_{1},\ \ldots ,\ {\mathrm{w}}_{\mathrm{\ell }}\right)$ would have also been a basis for $\ker⁡({\mathrm{T}}^{\prime }) .$ The second equality follows directly from part (c).

Suppose $T \in \mathscr{L}(V,W) .$ Then a) $T$ is injective $\Longleftrightarrow 0$ is not a singular value of $T^* T$ b) The number of positive singular values of $T$ equals $\dim{\left(T\left(V\right)\right)}.$ c) $T$ is surjective $\Longleftrightarrow$ the number of positive singular values of $T$ is $dim⁡(W) .$

a) Since $\ker{\left(T\right)}=\ker⁡(T^* T)$, we can conclude that $T$ is injective $⇔$ $T^* T$ is injective $\Longleftrightarrow 0$ is not an eigenvalue of $T^* T.$ b) Spectral theorem proves this: We have an orthogonal decomposition of eigenspaces and the kernel is just one of them (for $\lambda=0$). If you remove the basis vectors that are in the kernel, you have a basis for the range, and that number is the same as the number of positive singular values (remember that we have one sigular value for each dimension for its corresponding eigenspace). c) This follows from part (b): We just established that the range has the same dimension as this number. Now, that's equal to the dimension of W if and only if $T$ is surjective.

Suppose $S\in L\left(V,W\right).$ Then $S$ is an isometry $\Longleftrightarrow$ all of the singular values of $T^* T$ are $1.$

We begin by noting that $S$ is an isometry if and only if $S^*$ is its inverse, i.e. $S^* S=I .$ Then the result follows, since the singular values are the diagonal entries of the diagonalized $S^* S.$

Suppose $T∈L(V,W)$ and the positive singular values of $T$ are $s_{1},\ldots ,\ s_{m} .$ Then there exist orthonormal lists $(e_{1},\ldots ,\ e_{m})$ in $V$ and $(f_{1},\ldots ,\ f_{m})$ in $W$ such that $T\left(v\right)=s_{1}\left\langle v,\ e_{1}\right\rangle f_{1}+\ldots +s_{m}\left\langle v,e_{m}\right\rangle f_{m}$ for every $v \in V .$

The orthonormal basis $(e_{1},\ldots ,\ e_{m})$ can be taken as orthonormal bases for each eigenspace (or just apply spectral theorem) for the operator $T^* T .$ The orthonormal basis $(f_{1},\ldots , f_m)$ is found by scaling each $T(e_{i})$ by $\frac{1}{s_{i}} .$ This results in an orthonormal basis for $T(V)$ because these images are all orthogonal and their norms are their corresponding singular value squared. Here is the verification of that fact:

Suppose $T∈L(V,W)$ and the positive singular values of $T$ are $s_{1},\ldots ,\ s_{m} .$ Let $(e_{1},\ldots ,\ e_{m})$, and $\left(f_{1},\ \ldots ,\ f_{m}\right)$ be orthonormal lists in $V$ and $W$ respectively such that for every $v∈V$, $T\left(v\right)=s_{1}\left\langle v,e_{1}\right\rangle f_{1}+\ldots +s_{m}\left\langle v,e_{m}\right\rangle f_{m} .$ Then a)

b)

a) This is a formula for $T^* (w)$, so without effecting the outcome we can replace $w$ with $w$ projected onto $T(V)$, the orthogonal complement of $\ker⁡(T^* ) .$ Projecting $w$ this way makes this much easier to compute, since we have such a nice basis for $T^* T .$ Projected $w$ is $\left\langle w,f_{1}\right\rangle f_{1}+\ldots +\left\langle w,f_{m}\right\rangle f_{m}$, which is $T\left(\left\langle w,f_{1}\right\rangle e_{1}+\ldots +\left\langle w,f_{m}\right\rangle e_{m}\right) .$ Then $T^* \left(w\right)$ is the operator $T^* T$ applied to $\left\langle w,f_{1}\right\rangle e_{1}+\ldots +\left\langle w,f_{m}\right\rangle e_{m} .$ Now, $T^* T$ scales each basis vector by its respective $s_{i}^2$, we have

b) Let us start by retracing the same steps that we took for the last proof. $w$ was first projected onto the range, $T(V) .$ This is the first step in computing the pseudoinverse. Then applying $T^*$ gives us the result $s_{1}\left\langle w,f_{1}\right\rangle e_{1}+\ldots +s_{m}\left\langle w,f_{m}\right\rangle e_{m} .$ However, this is not something in the pre-image of our projected $w .$ It is the $v$ we are looking for after the application of $T^* T .$ But because $T^* T$ just scales the $i$th basis vector by $s_{i}^{2}$, we can divide each element by $s_{i}^{2}$ to obtain what we want. This is not the only pre-image vector for $s_{1}\left\langle w,f_{1}\right\rangle e_{1}+\ldots +s_{m}\left\langle w,f_{m}\right\rangle e_{m}$, but it is the one with the smallest norm because adding anything in the kernel of $T$, which is the orthogonal complement of the span of these basis vectors, would only increase the norm of the given vector. Therefore $T^{\dagger }\left(w\right)=\frac{\left\langle w,f_{1}\right\rangle }{s_{1}}e_{1}+\ldots +\frac{\left\langle w,f_{m}\right\rangle }{s_{m}}e_{m}.$

Suppose $A$ is an $M$-by-$n$ matrix of rank $m≥1 .$ Then there exist an $M$-by-$m$ matrix $B$ with orthonormal columns, an $m$-by-$m$ diagonal matrix $D$ with positive numbers on the diagonal, and an $n$-by-$m$ matrix $C$ with orthonormal columns such that $A=BDC^* .$

Let us observe what $BDC^*$ does to a basis vector $e_{i}$ from the SVD of $A$ as an operator, verify that it is sent to $f_{i}$, and then we will check that everything in the kernel of $A$, which is the orthogonal complement of the span of these $e_{i}$s, is sent to $0 .$ First, $e_{i}$ is sent by $C^*$ to the standard $i$th basis vector. This is sent to the $i$th column of $D$, which is $s_{i}$ times the $i$th standard basis vector by $D$, and then $B$ sends this to $s_{i}f_{i}$, since $f_{i}$ is the $i$th column of $B .$ Now, the kernel of $A$ is the orthogonal complement of the span of the $e_i$'s, so $C^*$ has columns all orthogonal to it and it is mapped to $0$ by $C^*.$