2.1.1 Neural networks for PCA

Principal components can be extracted using single-layer feed-forward neural networks (Sanger, 1989; Rubner and Tavan, 1989; Oja, 1989; Diamantaras and Kung, 1996). These networks learn unsupervised by using variants of the Hebbian rule. They provide an iterative solution to (2.5) and do not need the computation of $\bf C$, which could be computationally expensive (it is O(d ²n)). Networks extracting principal components further provide a biological basis for PCA. One example of such a PCA algorithm is Oja's rule.

Oja's algorithm (Oja, 1982) uses a single neuron with an input vector $\bf x$, a weight vector $\bf w$, and an output y. The output can be written as y = $\bf w^{T}_{}$$\bf x$ (this corresponds to (2.1)). According to Oja's rule, after a training pattern $\bf x$ is presented, the weights change by a Hebbian term minus a forgetting function:

$$\bf w \bf w \varepsilon \bf x \alpha \bf w$$ (2.6)

$\varepsilon$ is the Hebbian learning rate, and $\alpha$ is a constant. The forgetting term is necessary to bound the magnitude of $\bf w$. For the average update over all training patterns $\bf x_{i}^{}$, the fixed points of (2.6) can be computed. In turns out that they are the eigenvectors of the covariance matrix $\bf C$, and the eigenvector with the largest eigenvalue is the only stable point (Oja, 1982). Thus, Oja's rule extracts the principal component.