beamer-tu-logo

Lecture 6: Moments and moment generating functions

The various moments of a random variable are an important class of expectations.

Definition 2.3.1.

For each positive integer n, the nth moment of a random variable X (or F_{X}) is E(X^{n}). For each positive integer n, the nth central moment of X is E(X − μ)^{n}, where μ = E(X). For each constant p > 0, the pth absolute moment of X is E(|X|^{p}).

Property.

For p > 0 and any random variable X,

E|X|^{p} < ∞ iff ∑_{n=1}^{∞}n^{p−1} P(|X| ≥ n) < ∞

A consequence is, if the pth absolute moment (moment or central moment) exists and q < p, then the pth absolute moment (moment or central moment) also exists.

UW-Madison (Statistics)  Stat 609 Lecture 6  2014  1 / 11

beamer-tu-logo

Lecture 6: Moments and moment generating functions

The various moments of a random variable are an important class of expectations.

Definition 2.3.1.

For each positive integer n, the nth moment of a random variable X (or F_{X}) is E(X^{n}). For each positive integer n, the nth central moment of X is E(X − μ)^{n}, where μ = E(X). For each constant p > 0, the pth absolute moment of X is E(|X|^{p}).

Property.

For p > 0 and any random variable X,

E|X|^{p} < ∞ iff ∑_{n=1}^{∞}n^{p−1} P(|X| ≥ n) < ∞

A consequence is, if the pth absolute moment (moment or central moment) exists and q < p, then the pth absolute moment (moment or central moment) also exists.

UW-Madison (Statistics)  Stat 609 Lecture 6  2014  1 / 11

beamer-tu-logo

Proof.

From the discussion in the last lecture,

E|X|^{p} ≤ ∫_{0}^{∞} P(|X|^{p} ≥ x)dx = p ∫_{0}^{∞} P(|X|^{p} ≥ y^{p})y^{p−1} dy
= p ∑_{n=0}^{∞}∫_{n}^{n+1}P(|X|^{p} ≥ y^{p})y^{p−1} dy ≤ p ∑_{n=0}^{∞}(n + 1)^{p−1} P(|X| ≥ n)
≤ p + 2^{p−1} p ∑_{n=1}^{∞}n^{p−1} P(|X| ≥ n)

where the first equality is from changing variable x = y^{p}. Similarly,

E|X|^{p} ≥ ∫_{0}^{∞} P(|X|^{p} > x)dx = p ∑_{n=0}^{∞}∫_{n}^{n+1}P(|X|^{p} > y^{p})y^{p−1} dy
≥ p ∑_{n=0}^{∞}n^{p−1} P(|X| ≥ n + 1) = p ∑_{n=2}^{∞}(n − 1)^{p−1} P(|X| ≥ n)
≥ p2^{−p} ∑_{n=2}^{∞}n^{p−1} P(|X| ≥ n)
UW-Madison (Statistics)  Stat 609 Lecture 6  2014  2 / 11

beamer-tu-logo

Definition 2.3.2.

The second central moment of a random variable X is called its variance and denoted by Var(X) = E(X − μ)^{2}. The standard deviation of X is defined as \sqrt{Var(X)}.

Sometime the following result is useful:

Var(X) = E(X^{2}) − [E(X)]^{2}

Measures of spread

The variance gives a measure of the degree of spread of a distribution around its mean (center). A large variance means X is more variable. The unit on the variance is the square of the original unit. The standard deviation has the same qualitative interpretation as the variance, but its unit is the same as the original unit. If Var(X) = 0, then P(X = E(X)) = 1. This is actually a special case of the following result:

If g(X) ≥ 0 and E[g(X)] = 0, then P(g(X) = 0) = 1.

UW-Madison (Statistics)  Stat 609 Lecture 6  2014  3 / 11

beamer-tu-logo

Definition 2.3.2.

The second central moment of a random variable X is called its variance and denoted by Var(X) = E(X − μ)^{2}. The standard deviation of X is defined as \sqrt{Var(X)}.

Sometime the following result is useful:

Var(X) = E(X^{2}) − [E(X)]^{2}

Measures of spread

The variance gives a measure of the degree of spread of a distribution around its mean (center). A large variance means X is more variable. The unit on the variance is the square of the original unit. The standard deviation has the same qualitative interpretation as the variance, but its unit is the same as the original unit. If Var(X) = 0, then P(X = E(X)) = 1. This is actually a special case of the following result:

If g(X) ≥ 0 and E[g(X)] = 0, then P(g(X) = 0) = 1.

UW-Madison (Statistics)  Stat 609 Lecture 6  2014  3 / 11

beamer-tu-logo

The result can be proved in general, but we only consider the case where X has a pdf or pmf.

Since g(X) ≥ 0 and E[g(X)] = 0, X cannot have a pdf unless g(x) = 0 for all x.

If X is discrete and have positive probabilities p_{j}’s to take two values x_{j}’s, then

0 = E[g(X)] = ∑_{k} g(x_{k})p_{k} ≥ g(x_{j})p_{j}

implies that g(x_{j}) = 0 for all j, i.e., P(g(X) = 0) = 1.

Theorem 2.3.4.

If X has a finite variance, then for any constants a and b,

Var(aX + b) = a^{2}Var(X).

Proof.

By definition,

Var(aX + b) = E[(aX + b) − E(aX + b)]^{2} = E[aX − aE(X)]^{2}
= a^{2}E[X − E(X)]^{2} = a^{2}Var(X)
UW-Madison (Statistics)  Stat 609 Lecture 6  2014  4 / 11

beamer-tu-logo

The result can be proved in general, but we only consider the case where X has a pdf or pmf.

Since g(X) ≥ 0 and E[g(X)] = 0, X cannot have a pdf unless g(x) = 0 for all x.

If X is discrete and have positive probabilities p_{j}’s to take two values x_{j}’s, then

0 = E[g(X)] = ∑_{k} g(x_{k})p_{k} ≥ g(x_{j})p_{j}

implies that g(x_{j}) = 0 for all j, i.e., P(g(X) = 0) = 1.

Theorem 2.3.4.

If X has a finite variance, then for any constants a and b,

Var(aX + b) = a^{2}Var(X).

Proof.

By definition,

Var(aX + b) = E[(aX + b) − E(aX + b)]^{2} = E[aX − aE(X)]^{2}
= a^{2}E[X − E(X)]^{2} = a^{2}Var(X)
UW-Madison (Statistics)  Stat 609 Lecture 6  2014  4 / 11

beamer-tu-logo

Example 2.3.3 (the exponential distribution)

Suppose that X has the following pdf:

f_{X}(x) = \{ \begin{matrix}\frac{1}{λ}e^{−x/λ} & x ≥ 0 \\ 0 & x < 0 \end{matrix}

where λ > 0 is a constant.

μ = E(X) = ∫_{0}^{∞} \frac{1}{λ} xe^{−x/λ} dx = −xe^{−x/λ} |_{0}^{∞} + ∫_{0}^{∞} e^{−x/λ} dx = λ

Var(X) = E(X − λ)^{2} = E(X^{2}) − λ^{2} = ∫_{0}^{∞} \frac{x^{2}}{λ}e^{−x/λ} dx − λ^{2}
= −x^{2}e^{−x/λ} |_{0}^{∞} + ∫_{0}^{∞} 2xe^{−x/λ} dx − λ^{2}
= 2λ ∫_{0}^{∞} \frac{1}{λ} xe^{−x/λ} dx − λ^{2}
= 2λ^{2} − λ^{2} = λ^{2}
UW-Madison (Statistics)  Stat 609 Lecture 6  2014  5 / 11

beamer-tu-logo

Example 2.3.5 (binomial distribution)

Suppose that X is discrete with pmf

f_{X}(x) = \binom{n}{x} p^{x}(1 − p)^{n−x} , x = 0,1,...,n,

where n is a fixed positive integer and 0 < p < 1 is a fixed constant. We have obtained that E(X) = np. Using the identity

x^{2}\binom{n}{x} = x \frac{n!}{(x − 1)!(n − x)!} = xn\binom{n − 1}{x − 1}

we obtain

E(X^{2}) = ∑_{x=0}^{n}x^{2}\binom{n}{x} p^{x}(1 − p)^{n−p} = n ∑_{x=1}^{n}x\binom{n − 1}{x − 1} p^{x}(1 − p)^{n−x}

= np ∑_{y=0}^{n−1}(y + 1)\binom{n − 1}{y} p^{y}(1 − p)^{n−1−y}
= np[(n − 1)p + 1] = n(n − 1)p^{2} + np

Var(X) = E(X^{2}) − (EX)^{2} = n(n − 1)p^{2} + np − (np)^{2} = np(1 − p).

UW-Madison (Statistics)  Stat 609 Lecture 6  2014  6 / 11

beamer-tu-logo

Definition 2.3.6 (moment generating function).

The moment generating function (mgf) of a random variable X is

M_{X}(t) = E(e^{tX}) = \{ \begin{matrix}∑_{x} e^{tx}f_{X}(x) & if X has a pmf \\ ∫_{−∞}^{∞} e^{tx}f_{X}(x)dx & if X has a pdf \end{matrix}

provided that E(e^{tX}) exists. (Note that M_{X}(0) = E(e^{0X}) = 1 always exists.) Otherwise, we say that the mgf M_{X}(t) does not exist at t.

Theorem 2.3.15.

For any constants a and b, the mgf of the random variable aX + b is

M_{aX+b}(t) = e^{bt}M_{X}(at)

If M_{X}(t) exists in a neighborhood of 0, then M_{aX+b}(t) exists in a neighborhood of 0.

Proof.

By definition,

M_{aX+b}(t) = E(e^{t(aX+b)}) = E(e^{taX}e^{bt}) = e^{bt}E(e^{(ta)X}) = e^{bt}M_{X}(at)

UW-Madison (Statistics)  Stat 609 Lecture 6  2014  7 / 11

beamer-tu-logo

Definition 2.3.6 (moment generating function).

The moment generating function (mgf) of a random variable X is

M_{X}(t) = E(e^{tX}) = \{ \begin{matrix}∑_{x} e^{tx}f_{X}(x) & if X has a pmf \\ ∫_{−∞}^{∞} e^{tx}f_{X}(x)dx & if X has a pdf \end{matrix}

provided that E(e^{tX}) exists. (Note that M_{X}(0) = E(e^{0X}) = 1 always exists.) Otherwise, we say that the mgf M_{X}(t) does not exist at t.

Theorem 2.3.15.

For any constants a and b, the mgf of the random variable aX + b is

M_{aX+b}(t) = e^{bt}M_{X}(at)

If M_{X}(t) exists in a neighborhood of 0, then M_{aX+b}(t) exists in a neighborhood of 0.

Proof.

By definition,

M_{aX+b}(t) = E(e^{t(aX+b)}) = E(e^{taX}e^{bt}) = e^{bt}E(e^{(ta)X}) = e^{bt}M_{X}(at)

UW-Madison (Statistics)  Stat 609 Lecture 6  2014  7 / 11

beamer-tu-logo

The main use of mgf

It can be used to generate moments (discussed in this lecture). It helps to characterize a distribution (discussed in next lecture).

Theorem 2.3.7.

If M_{X}(t) exists at ±t, then E(X^{n}) exists for any positive integer n and

E(X^{n}) = M_{X}^{(n)}(0) = \frac{d^{n}}{dt^{n}}M_{X}(t)|_{t=0}

i.e., the nth moment is the nth derivative of M_{X}(t) evaluated at t = 0.

Proof.

Assuming that we can exchange the differentiation and integration (which will be justified later),

\frac{d^{n}}{dt^{n}}M_{X}(t) = \frac{d^{n}}{dt^{n}}∫_{−∞}^{∞} e^{tx}f_{X}(x)dx = ∫_{−∞}^{∞} \frac{d^{n}e^{tx}}{dt^{n}} f_{X}(x)dx = ∫_{−∞}^{∞} x^{n}e^{tx}f_{X}(x)dx

Hence

\frac{d^{n}}{dt^{n}}M_{X}(t)|_{t=0} = ∫_{−∞}^{∞} x^{n}e^{0x}f_{X}(x)dx = ∫_{−∞}^{∞} x^{n}f_{X}(x)dx = E(X^{n})
UW-Madison (Statistics)  Stat 609 Lecture 6  2014  8 / 11

beamer-tu-logo

The main use of mgf

It can be used to generate moments (discussed in this lecture). It helps to characterize a distribution (discussed in next lecture).

Theorem 2.3.7.

If M_{X}(t) exists at ±t, then E(X^{n}) exists for any positive integer n and

E(X^{n}) = M_{X}^{(n)}(0) = \frac{d^{n}}{dt^{n}}M_{X}(t)|_{t=0}

i.e., the nth moment is the nth derivative of M_{X}(t) evaluated at t = 0.

Proof.

Assuming that we can exchange the differentiation and integration (which will be justified later),

\frac{d^{n}}{dt^{n}}M_{X}(t) = \frac{d^{n}}{dt^{n}}∫_{−∞}^{∞} e^{tx}f_{X}(x)dx = ∫_{−∞}^{∞} \frac{d^{n}e^{tx}}{dt^{n}} f_{X}(x)dx = ∫_{−∞}^{∞} x^{n}e^{tx}f_{X}(x)dx

Hence

\frac{d^{n}}{dt^{n}}M_{X}(t)|_{t=0} = ∫_{−∞}^{∞} x^{n}e^{0x}f_{X}(x)dx = ∫_{−∞}^{∞} x^{n}f_{X}(x)dx = E(X^{n})
UW-Madison (Statistics)  Stat 609 Lecture 6  2014  8 / 11

beamer-tu-logo

The condition that M_{X}(t) exists at ±t is in fact used to ensure the validity of the exchange of differentiation and integration. Besides this, we also need to argue that E|X|^{n} < ∞ for any positive integer n under the condition that M_{X}(t) exists at ±t. We first show that, if M_{X}(t) exists at ±t, then for any s ∈ (−t,t), M_{X}(s) exists:

M_{X}(s) = E(e^{sX}) = E[e^{sX}I(X > 0)] + E[e^{sX}I(X ≤ 0)]
≤ E[e^{tX}I(X > 0)] + E[e^{−tX}I(X ≤ 0)] ≤ E(e^{tX}) + E(e^{−tX})
= M_{X}(t) + M_{X}(−t) < ∞

where I(X > 0) is the indicator of X > 0. Next, we show that, for any positive p > 0, E|X|^{p} < ∞ under the condition M_{X}(t) exists at ±t. For a given p > 0, choose s such that 0 < ps < t. Because s|X| ≤ e^{s|X|}, we have s^{p}|X|^{p} ≤ e^{ps|X|} ≤ e^{psX} + e^{−psX} and

E|X|^{p} ≤ s^{−p} E(e^{psX} + e^{−psX} ) = s^{−p} M_{X}(ps) + M_{X}(−ps) < ∞

UW-Madison (Statistics)  Stat 609 Lecture 6  2014  9 / 11

beamer-tu-logo

In fact, the condition that M_{X}(t) exists at ±t ensures that M_{X}(s) has the power series expansion

M_{X}(s) = ∑_{k=0}^{∞} \frac{E(X^{k})s^{k}}{k!} − t < s < t

If the distribution of X is symmetric (about 0), i.e., X and −X have the same distribution, then

M_{X}(t) = E(e^{tX}) = E(e^{t(−X)}) = E(e^{−tX}) = M_{X}(−t)

i.e., M_{X}(t) is an even function and M_{X}(t) exists at ±t is the same as M_{X}(t) exists at a t > 0.

Example 2.3.8 (Gamma mgf)

Let X have the gamma pdf

f_{X}(x) = \frac{1}{Γ(α)β^{α}} x^{α−1} e^{−x/β} , x > 0,

where α > 0 and β > 0 are two constants and

Γ(α) = ∫_{0}^{∞} x^{α−1} e^{−x} dx α > 0

beamer-tu-logo

In fact, the condition that M_{X}(t) exists at ±t ensures that M_{X}(s) has the power series expansion

M_{X}(s) = ∑_{k=0}^{∞} \frac{E(X^{k})s^{k}}{k!} − t < s < t

If the distribution of X is symmetric (about 0), i.e., X and −X have the same distribution, then

M_{X}(t) = E(e^{tX}) = E(e^{t(−X)}) = E(e^{−tX}) = M_{X}(−t)

i.e., M_{X}(t) is an even function and M_{X}(t) exists at ±t is the same as M_{X}(t) exists at a t > 0.

Example 2.3.8 (Gamma mgf)

Let X have the gamma pdf

f_{X}(x) = \frac{1}{Γ(α)β^{α}} x^{α−1} e^{−x/β} , x > 0,

where α > 0 and β > 0 are two constants and

Γ(α) = ∫_{0}^{∞} x^{α−1} e^{−x} dx α > 0

beamer-tu-logo

If t < 1/β,

M_{X}(t) = E(e^{tX}) = \frac{1}{Γ(α)β^{α}} ∫_{0}^{∞} e^{tx}x^{α−1} e^{−x/β} dx
= \frac{1}{Γ(α)β^{α}} ∫_{0}^{∞} x^{α−1} e^{−x/( \frac{β}{1−βt} )} dx
= \frac{(\frac{β}{1−βt} )^{α}}{Γ(α)β^{α}} ∫_{0}^{∞} s^{α−1} e^{−s} ds = \frac{1}{(1 − βt)^{α}}

If t ≥ 1/β, then E(e^{tX}) = ∞. We can obtain

E(X) = \frac{d}{dt}M_{X}(t)|_{t=0} = \frac{αβ}{(1 − βt)^{α+1}} |_{t=0} = αβ

For any integer n > 1,

E(X^{n}) = \frac{d^{n}}{dt^{n}}M_{X}(t)|_{t=0} = \frac{α(α + 1)··· (α + n − 1)β^{n}}{(1 − βt)^{α+n}} |_{t=0}
= α(α + 1)··· (α + n − 1)β^{n}

UW-Madison (Statistics)  Stat 609 Lecture 6  2014  11 / 11