2.1 Principal component analysis (PCA)

Principal component analysis is a widely used tool for dimension reduction (Diamantaras and Kung, 1996). Let $\bf x_{i}^{}$ $\in$ IR^d, where i = 1,..., n, be the training patterns. The principal components are a set of q < d orthonormal vectors and span a subspace in the major directions into which the patterns extend (figure 2.1).

Figure 2.1: The principal component $\bf w_{1}^{}$ points into the direction of maximum variance. The gray dots are the training patterns. The intersection of the dashed lines is the center of the pattern distribution.

In this section, we assume that the patterns are centered around the origin (without loss of generality). Let $\bf y$ be the projection onto a subspace,

$$\bf y \bf W^{T}_{} \bf x$$ (2.1)

$\bf W$ is a d×q matrix that contains the principal components as columns. The vector $\bf y$ is a dimension-reduced representation of $\bf x$. Let $\hat{{\bf x}}$ be the reconstruction of $\bf x$ given only the vector $\bf y$,

$$\hat{{\bf x}} \bf W \bf y$$ (2.2)

The goal of PCA is to set the subspace such that the mean reconstruction error E_rec is minimized,

$${\frac{{1}}{{n}}} \sum_{{i=1}}^{n} \left\Vert\vphantom{{\bf x}_i - \hat{\bf x}_i}\right. \bf x_{i}^{} \hat{{\bf x}}_{i}^{} \left.\vphantom{{\bf x}_i - \hat{\bf x}_i}\right\Vert^{2}_{}$$ (2.3)

This goal is equivalent to finding the q major directions of maximal variance within the set of patterns {$\bf x_{i}^{}$} (Diamantaras and Kung, 1996). Moreover, it is equivalent to the principal components being the first q eigenvectors $\bf w_{l}^{}$ of the covariance matrix $\bf C$ of the pattern set (Diamantaras and Kung, 1996),

$$\bf C {\frac{{1}}{{n}}} \sum_{{i=1}}^{n} \bf x_{i}^{} \bf x_{i}^{T}$$ (2.4)

The corresponding eigenvalue equation is

$$\bf C \bf w_{l}^{} \lambda_{l}^{} \bf w_{l}^{}$$ (2.5)

The eigenvalue $\lambda_{l}^{}$ is the variance of the distribution {$\bf x_{i}^{}$} in the direction of $\bf w_{l}^{}$. The following sections describe how neural networks can extract principal components and how PCA can be linked to the probability density of a pattern distribution.