Fisher's linear discriminant

Fisher's linear discriminant [1]is a quantity which is derived to solve binary supervised classification problems (see Classifiers_and_Discriminants).

It is assumed that every object or event $i$

belongs to one of two classes $y_{i}\in \{-1,1\}$

, and may be represented by a vector of measurements referred to as a feature vector $x_i$ . Since the measurements have been chosen to distinguish between the classes, the distribution of each class in this feature space should be different. For simplicity the distributions of each class $k$

have been summarised by their means $\mu_k$
and covariances $\Sigma _{k}$

.

Fisher's linear discriminant is a linear combination of the feature vectors $x_i$

which maximises the separation of the two classes in one quantity. In general, two classes will be further apart when their means are far apart, and there is little variability within each class. Fisher summarised this with the measure of separation given by

Separation = ${\frac {Between\;class\;variance}{Within\;class\;variance}}$

For this example, consider the linear combination of measurements

$\sum _{i}n_{i}x_{i}$

It can be shown that the mean of each class in this new variable will be $n^{T}\mu _{1}$

and $n^{T}\mu _{2}$

, and the variances will be $n^{T}\Sigma _{1}n$

and $n^{T}\Sigma _{2}n$

. The Fisher separation will then be

Separation = ${\frac {n^{T}(\mu _{2}-\mu _{1})}{n^{T}(\Sigma _{1}+\Sigma _{2})n}}$

which should be maximised with respect to $n$ . It can be shown that the optimal $n$

is given by

$n=(\Sigma _{1}+\Sigma _{2})^{-1}(\mu _{2}-\mu _{1})$

and so Fisher's linear discriminant is given by $n^{T}x$ . A set of measurements can then be classified by a simple comparison of the Fisher discriminant with a threshold.

It is frequently stated that Fisher's discriminant assumes each class is normally distributed. This is probably due to confusion with the related Linear Discriminant Analysis (LDA), which is in fact identical to Fisher's discriminant when the covariance of both classes are equal.

[1] ``The use of multiple measurements in taxonomic problems," R.A.Fisher, Annals of Eugenics, Vol.7, pp.179--188, 1936.