A.2 Maximum likelihood

The maximum likelihood principle is illustrated in an example with a one-dimensional data distribution {x_i}, i = 1,..., n. We assume that the data originate from a Gaussian distribution p(x) with parameters $\sigma$ and $\mu$,

$${\frac{{1}}{{\sqrt{2 \pi} \sigma}}} \left(\vphantom{-\frac{(x-\mu)^2}{2 \sigma^2}}\right. {\frac{{(x-\mu)^2}}{{2 \sigma^2}}} \left.\vphantom{-\frac{(x-\mu)^2}{2 \sigma^2}}\right)$$ (A.3)

According to the maximum likelihood principle, we will choose the unknown parameters such that the given data are most likely under the obtained distribution. The probability L of the given data set is

$$\sigma \mu \prod^{n}_{{i=1}} \left(\vphantom{\frac{1}{\sqrt{2 \pi} \sigma}}\right. {\frac{{1}}{{\sqrt{2 \pi} \sigma}}} \left.\vphantom{\frac{1}{\sqrt{2 \pi} \sigma}}\right)^{n}_{} \left(\vphantom{-\frac{\sum_{i=1}^n(x_i-\mu)^2}{2 \sigma^2}}\right. {\frac{{\sum_{i=1}^n(x_i-\mu)^2}}{{2 \sigma^2}}} \left.\vphantom{-\frac{\sum_{i=1}^n(x_i-\mu)^2}{2 \sigma^2}}\right)$$ (A.4)

We want to find $\hat{{\sigma}}$ and $\hat{{\mu}}$ that maximize L. Maximizing L is equivalent to maximizing log L, which is also called the log-likelihood $\mathcal {L}$,

$$\mathcal {L} \sigma \mu \sigma \mu \sigma {\frac{{\sum_i(x_i-\mu)^2}}{{2 \sigma^2}}}$$ (A.5)

To find the maximum we compute the derivatives of the log-likelihood $\mathcal {L}$ and set them to zero:

$${\frac{{\partial\mathcal{L}}}{{\partial\sigma}}} {\frac{{n}}{{\sigma}}} {\frac{{\sum_i(x_i-\mu)^2}}{{\sigma^3}}} \;\stackrel{{\textstyle!}}{{=}}\;$$ = (A.6)

$${\frac{{\partial\mathcal{L}}}{{\partial\mu}}} {\frac{{\sum_i(x_i-\mu)}}{{\sigma^2}}} \;\stackrel{{\textstyle!}}{{=}}\;$$ = (A.7)

Thus, we obtain the values of the parameters $\hat{{\sigma}}$ and $\hat{{\mu}}$:

$$\hat{{\sigma}}^{2}_{} {\frac{{\sum_i(x_i-\hat{\mu})^2}}{{n}}}$$ = (A.8)

$$\hat{{\mu}} {\frac{{\sum_i x_i}}{{n}}}$$ = (A.9)

The resulting $\hat{\sigma}^{2}_{}$ is the variance of the distribution and $\hat{{\mu}}$ is its center. The extremum of $\mathcal {L}$ is indeed a local maximum, as can be seen by computing the Hesse matrix of $\mathcal {L}$ and evaluating it at the extreme point ($\hat{{\sigma}}$,$\hat{{\mu}}$):

$$\mathcal {L} \left\vert\vphantom{ \begin{array}{cc} \frac{\partial^2\mathcal{L... ...rtial\sigma} & \frac{\partial^2\mathcal{L}}{\partial\mu^2} \end{array} }\right. \begin{array}{cc} \frac{\partial^2\mathcal{L}}{\partial\sigma^2} ... ...ial\mu\partial\sigma} & \frac{\partial^2\mathcal{L}}{\partial\mu^2} \end{array} \left.\vphantom{ \begin{array}{cc} \frac{\partial^2\mathcal{L}}{\... ...l\sigma} & \frac{\partial^2\mathcal{L}}{\partial\mu^2} \end{array} }\right\vert$$ (A.10)

$${\frac{{\partial^2\mathcal{L}}}{{\partial\sigma^2}}} \Big\vert _{{\sigma=\hat{\sigma}, \mu=\hat{\mu}}}^{} {\frac{{n}}{{\hat{\sigma}^2}}} {\frac{{3\sum_i (x_i-\hat{\mu})^2}}{{\hat{\sigma}^4}}} {\frac{{n}}{{\hat{\sigma}^2}}} {\frac{{3 n\hat{\sigma}^2}}{{\hat{\sigma}^4}}} {\frac{{2 n}}{{\hat{\sigma}^2}}}$$ = (A.11)

$${\frac{{\partial^2\mathcal{L}}}{{\partial\sigma\partial\mu}}} \Big\vert _{{\sigma=\hat{\sigma}, \mu=\hat{\mu}}}^{} {\frac{{\partial^2\mathcal{L}}}{{\partial\mu\partial\sigma}}} \Big\vert _{{\sigma=\hat{\sigma}, \mu=\hat{\mu}}}^{} {\frac{{2 \sum_i(x_i - \hat{\mu})}}{{\hat{\sigma}^3}}}$$ =

$${\frac{{\partial^2\mathcal{L}}}{{\partial\mu^2}}} \Big\vert _{{\sigma=\hat{\sigma}, \mu=\hat{\mu}}}^{} {\frac{{n}}{{\hat{\sigma}^2}}}$$ =

It follows that the Hesse matrix at the extremum is negative definite,

$$\mathcal {L} \sigma \hat{{\sigma}} \mu \hat{{\mu}} \left\vert\vphantom{ \begin{array}{cc} - \frac{2 n}{\hat{\sigma}^2} & 0 \ \ 0 & - \frac{n}{\hat{\sigma}^2} \end{array} }\right. \begin{array}{cc} - \frac{2 n}{\hat{\sigma}^2} & 0 \ \ 0 & - \frac{n}{\hat{\sigma}^2} \end{array} \left.\vphantom{ \begin{array}{cc} - \frac{2 n}{\hat{\sigma}^2} & 0 \ \ 0 & - \frac{n}{\hat{\sigma}^2} \end{array} }\right\vert$$ (A.12)

Therefore, the extremum is a local maximum. Moreover, it is also a global maximum. First, for finite parameters, no other extrema exist because $\mathcal {L}$ is a smooth function. Second, $\mathcal {L}$ is positive for finite parameters, but approaches zero for infinite values. Thus, any maximum must be in the finite range.