Page 308 - 35Linear Algebra

308 Least squares and Singular Values

17.2 Singular Value Decomposition

Suppose
linear
L : V −−−→W .
It is unlikely that dim V =: n = m := dim W so a m × n matrix M of L
in bases for V and W will not be square. Therefore there is no eigenvalue
problem we can use to uncover a preferred basis. However, if the vector
spaces V and W both have inner products, there does exist an analog of the
eigenvalue problem, namely the singular values of L.
Before giving the details of the powerful technique known as the singular
value decomposition, we note that it is an excellent example of what Eugene
Wigner called the “Unreasonable Effectiveness of Mathematics”:

There is a story about two friends who were classmates in high school, talking about
their jobs. One of them became a statistician and was working on population trends. He
showed a reprint to his former classmate. The reprint started, as usual with the Gaussian
distribution and the statistician explained to his former classmate the meaning of the
symbols for the actual population and so on. His classmate was a bit incredulous and was
not quite sure whether the statistician was pulling his leg. “How can you know that?”
was his query. “And what is this symbol here?” “Oh,” said the statistician, this is “π.”
“And what is that?” “The ratio of the circumference of the circle to its diameter.” “Well,
now you are pushing your joke too far,” said the classmate, “surely the population has
nothing to do with the circumference of the circle.”
Eugene Wigner, Commun. Pure and Appl. Math. XIII, 1 (1960).

Whenever we mathematically model a system, any “canonical quantities”
(those that do not depend on any choices we make for calculating them) will
correspond to important features of the system. For examples, the eigenval-
ues of the eigenvector equation you found in review question 1, chapter 12
encode the notes and harmonics that a guitar string can play!
Singular values appear in many linear algebra applications, especially
those involving very large data sets such as statistics and signal processing.
Let us focus on the m×n matrix M of a linear transformation L : V → W
written in orthonormal bases for the input and outputs of L (notice, the
existence of these othonormal bases is predicated on having inner products for
V and W). Even though the matrix M is not square, both the matrices MM T
T
and M M are square and symmetric! In terms of linear transformations M T
is the matrix of a linear transformation

linear
∗
L : W−−−→V .

308