Q. What is maximum likelihood estimation (MLE)?

Maximum Likelihood Estimation (MLE) is a method of parameter estimation in statistics and is an indispensable tool for many statistical modeling techniques, in particular in non-linear modeling with non-normal data. In essence, MLE is a method to seek the probability distribution that makes the observed data most likely.

Optimal properties in estimation:

1. Sufficiency - complete information about the parameter of interest contained in its MLE estimator
2. Consistency - true parameter value that generated data asymptotically) data of sufficiently large samples)
3. Efficiency - lowest-possible variance of parameter estimates achieved asymptotically
4. Parameterization invariance - same MLE solution obtained independent of the parametrization used)

Definitions:

Let X₁, X₂,..., X_n be a random sample from a distribution that depends on one or more unknown parameters θ₁, θ₂,..., θ_m with probability density (or mass) function f(x_i; θ₁, θ₂,..., θ_m). Suppose that (θ₁, θ₂,..., θ_m) is restricted to a given parameter space Ω. Then:

(1) When regarded as a function of θ₁, θ₂,..., θ_m, the joint probability density (or mass) function of X₁, X₂,..., X_n:

L(θ1,θ2,…,θm)=∏i=1nf(xi;θ1,θ2,…,θm)L(θ1,θ2,…,θm)=∏i=1nf(xi;θ1,θ2,…,θm)

((θ₁, θ₂,..., θ_m) in Ω) is called the likelihood function.

(2) If:

[u1(x1,x2,…,xn),u2(x1,x2,…,xn),…,um(x1,x2,…,xn)][u1(x1,x2,…,xn),u2(x1,x2,…,xn),…,um(x1,x2,…,xn)]

is the m-tuple that maximizes the likelihood function, then:

θ^i=ui(X1,X2,…,Xn)θ^i=ui(X1,X2,…,Xn)

is the maximum likelihood estimator of θ_i, for i = 1, 2, ..., m.

(3) The corresponding observed values of the statistics in (2), namely:

[u1(x1,x2,…,xn),u2(x1,x2,…,xn),…,um(x1,x2,…,xn)][u1(x1,x2,…,xn),u2(x1,x2,…,xn),…,um(x1,x2,…,xn)]

are called the maximum likelihood estimates of θ_i, for i = 1, 2, ..., m.

How to perform MLE?

Get the probability density function of the variables --> get the likelihood function --> get the MLE