Maximum Likelihood Estimation (MLE) for Multiple Regression

II.II.2 Maximum Likelihood Estimation (MLE) for Multiple Regression

MLE is needed when one introduces the following assumptions

(II.II.2-1)

(in this work we only focus on the use of MLE in cases where y and e are normally distributed).

The pdf of y is given by

(II.II.2-2)

and the log likelihood function

(II.II.2-3)

Maximizing the log likelihood function is (here) equivalent to minimizing the SSR

(II.II.2-4)

Therefore it follows

(II.II.2-5)

and of course

(II.II.2-6)

so that the ML estimator of ß is a BLUE and also a best unbiased estimator (for nonlinear models).

The ML estimator for the variance parameter however is biased

(II.II.2-7)

so that we have to use

(II.II.2-8)

The large sample properties of the ML estimator can be deduced on using a Taylor expansion of the likelihood around the true parameter value

(II.II.2-9)

This expression can be shown to be true under the so-called regularity condition which implies that the information matrix times 1/T converges to a probability matrix in the limit

(II.II.2-10)

Now it follows from (II.II.2-9) that

(II.II.2-11)

From (II.II.2-1)it can be seen that y is identically and independently distributed. Thus the pdf can be written as a derivative of the log likelihood

(II.II.2-12)

and from the derivation (c.q. proof) of the Cramér-Rao lower bound it follows that each of the T observations has a zero expected value and finite variance. Therefore

(II.II.2-13)

According to (II.II.2-10) and the central limit theorem it can be shown that

(II.II.2-14)

Using (II.II.2-11) and (II.II.2-14) it is easily derived that

(II.II.2-15)

Applying Cramér's theorem (I.VI-36) and (I.VI-37), proves that

(II.II.2-16)

(Q.E.D.) since

(II.II.2-17)