Page 308 - 35Linear Algebra
P. 308

308                                                          Least squares and Singular Values


                            17.2      Singular Value Decomposition


                            Suppose
                                                                 linear
                                                           L : V −−−→W .
                            It is unlikely that dim V =: n = m := dim W so a m × n matrix M of L
                            in bases for V and W will not be square. Therefore there is no eigenvalue
                            problem we can use to uncover a preferred basis. However, if the vector
                            spaces V and W both have inner products, there does exist an analog of the
                            eigenvalue problem, namely the singular values of L.
                               Before giving the details of the powerful technique known as the singular
                            value decomposition, we note that it is an excellent example of what Eugene
                            Wigner called the “Unreasonable Effectiveness of Mathematics”:

                                  There is a story about two friends who were classmates in high school, talking about
                                  their jobs. One of them became a statistician and was working on population trends. He
                                  showed a reprint to his former classmate. The reprint started, as usual with the Gaussian
                                  distribution and the statistician explained to his former classmate the meaning of the
                                  symbols for the actual population and so on. His classmate was a bit incredulous and was
                                  not quite sure whether the statistician was pulling his leg. “How can you know that?”
                                  was his query. “And what is this symbol here?” “Oh,” said the statistician, this is “π.”
                                  “And what is that?” “The ratio of the circumference of the circle to its diameter.” “Well,
                                  now you are pushing your joke too far,” said the classmate, “surely the population has
                                  nothing to do with the circumference of the circle.”
                                  Eugene Wigner, Commun. Pure and Appl. Math. XIII, 1 (1960).

                            Whenever we mathematically model a system, any “canonical quantities”
                            (those that do not depend on any choices we make for calculating them) will
                            correspond to important features of the system. For examples, the eigenval-
                            ues of the eigenvector equation you found in review question 1, chapter 12
                            encode the notes and harmonics that a guitar string can play!
                               Singular values appear in many linear algebra applications, especially
                            those involving very large data sets such as statistics and signal processing.
                               Let us focus on the m×n matrix M of a linear transformation L : V → W
                            written in orthonormal bases for the input and outputs of L (notice, the
                            existence of these othonormal bases is predicated on having inner products for
                            V and W). Even though the matrix M is not square, both the matrices MM      T
                                   T
                            and M M are square and symmetric! In terms of linear transformations M      T
                            is the matrix of a linear transformation

                                                                   linear
                                                            ∗
                                                           L : W−−−→V .

                                                      308
   303   304   305   306   307   308   309   310   311   312   313