C.3 Estimate of error accumulation

This section shows that the expectation value of the square error of the anticipated sensory input increases only linearly with the number of anticipation steps (Hoffmann and Möller, 2004). Let $\bf e$ be the error of the feed-forward output after a single step. $\bf e$ is a vector with one component for each output component. We assume that the probability distribution of this error is independent of the input to the network. Thus, all errors are independent of each other. In addition, we assume that the error for each output component has zero mean and the same standard deviation $\sigma$.

On this basis, we compute the expectation value of the square error. The total error of the chain output is the sum of the errors of the outputs of each link. To illustrate this, think of each correct transformation at one link as a line in a d-dimensional space, with d equal to the number of output components (figure C.1).

Figure C.1: Error accumulation in a feed-forward chain. Each solid black line is the correct transformation for one link. A dashed line is the correct transformation for a slightly different starting point.

A line connects an input point with an output point (of the transformation). The error at link i can be drawn as an arrow $\bf e_{i}^{}$ at the end of a line (output point). This will result in a different starting point for the next line. If the error is small^C.1 and the transformation function sufficiently smooth, we can approximate that the displacement of the starting point does not change the direction and length of the next line, which is the correct transformation at the new starting point. Thus, the displacement $\bf E$ of the final point is the sum of the vectorial errors of each stage. Therefore, given l links, the total error E can be written as

$$\left\Vert\vphantom{\sum_{i=1}^l {\bf e}_i }\right. \sum_{{i=1}}^{l} \bf e_{i}^{} \left.\vphantom{\sum_{i=1}^l {\bf e}_i }\right\Vert$$ (C.10)

We compute the expectation value of E²,

$$\left\langle\vphantom{ E^2 }\right. \left.\vphantom{ E^2 }\right\rangle \left\langle\vphantom{ \left\Vert\sum_{i=1}^l {\bf e}_i\right\Vert^2 }\right. \left\Vert\vphantom{\sum_{i=1}^l {\bf e}_i }\right. \sum_{{i=1}}^{l} \bf e_{i}^{} \left.\vphantom{\sum_{i=1}^l {\bf e}_i}\right\Vert^{2}_{} \left.\vphantom{ \left\Vert\sum_{i=1}^l {\bf e}_i\right\Vert^2 }\right\rangle$$ (C.11)

Doing the square operation on the sum gives

$$\left\langle\vphantom{ E^2 }\right. \left.\vphantom{ E^2 }\right\rangle \left\langle\vphantom{ \sum_i {\bf e}_i^T{\bf e}_i + \sum_{i,j\ne i} {\bf e}_i^T{\bf e}_j }\right. \sum_{i}^{} \bf e_{i}^{T} \bf e_{i}^{} \sum_{{i,j\ne i}}^{} \bf e_{i}^{T} \bf e_{j}^{} \left.\vphantom{ \sum_i {\bf e}_i^T{\bf e}_i + \sum_{i,j\ne i} {\bf e}_i^T{\bf e}_j }\right\rangle$$ (C.12)

and using the linear property of the expectation value results in

$$\left\langle\vphantom{ E^2 }\right. \left.\vphantom{ E^2 }\right\rangle \sum_{i}^{} \left\langle\vphantom{ {\bf e}_i^T{\bf e}_i }\right. \bf e_{i}^{T} \bf e_{i}^{} \left.\vphantom{ {\bf e}_i^T{\bf e}_i }\right\rangle \sum_{{i,j\ne i}}^{} \left\langle\vphantom{ {\bf e}_i^T{\bf e}_j }\right. \bf e_{i}^{T} \bf e_{j}^{} \left.\vphantom{ {\bf e}_i^T{\bf e}_j }\right\rangle$$ =

$$\sum_{i}^{} \left\langle\vphantom{ {\bf e}_i^T{\bf e}_i }\right. \bf e_{i}^{T} \bf e_{i}^{} \left.\vphantom{ {\bf e}_i^T{\bf e}_i }\right\rangle$$ = (C.13)

The last term vanishes because $\bf e_{i}^{}$ and $\bf e_{j}^{}$ are independent random variables, for i$\ne$j, and each variable has zero mean. The remainder is a sum over the variances for each link and dimension. Therefore,

$$\left\langle\vphantom{ E^2 }\right. \left.\vphantom{ E^2 }\right\rangle \sigma^{2}_{}$$ (C.14)

Thus, the expectation value of the square error increases only linearly with the chain length.