beamer-tu-logo

Lecture 7: Characterizing distributions by mgf’s

Can the moments determine a distribution?

Can two random variables with different distributions have the same moments of any order?

Example 2.3.10.

X_{1} has pdf  f_{1}(x) = \frac{1}{\sqrt{2π}x}e^{−(logx)^{2}/2} ,  x ≥ 0
X_{2} has pdf f_{2}(x) = f_{1}(x)[1 + sin(2π logx)], x ≥ 0

For any positive integer n,

E(X_{1}^{n}) = \frac{1}{\sqrt{2π}} ∫_{0}^{∞} x^{n−1} e^{−(logx)^{2}/2} dx
= \frac{1}{\sqrt{2π}} ∫_{−∞}^{∞} e^{ny−y^{2}/2} dy  y = logx
= \frac{e^{n^{2}/2}}{\sqrt{2π}} ∫_{−∞}^{∞} e^{−(y−n)^{2}/2} dy
= e^{n^{2}/2}  using the property of a normal distribution
UW-Madison (Statistics)  Stat 609 Lecture 7  2014  1 / 11

beamer-tu-logo

Lecture 7: Characterizing distributions by mgf’s

Can the moments determine a distribution?

Can two random variables with different distributions have the same moments of any order?

Example 2.3.10.

X_{1} has pdf  f_{1}(x) = \frac{1}{\sqrt{2π}x}e^{−(logx)^{2}/2} ,  x ≥ 0
X_{2} has pdf f_{2}(x) = f_{1}(x)[1 + sin(2π logx)], x ≥ 0

For any positive integer n,

E(X_{1}^{n}) = \frac{1}{\sqrt{2π}} ∫_{0}^{∞} x^{n−1} e^{−(logx)^{2}/2} dx
= \frac{1}{\sqrt{2π}} ∫_{−∞}^{∞} e^{ny−y^{2}/2} dy  y = logx
= \frac{e^{n^{2}/2}}{\sqrt{2π}} ∫_{−∞}^{∞} e^{−(y−n)^{2}/2} dy
= e^{n^{2}/2}  using the property of a normal distribution
UW-Madison (Statistics)  Stat 609 Lecture 7  2014  1 / 11

beamer-tu-logo

E(X_{2}^{n}) = ∫_{0}^{∞} x^{n}f_{1}(x)[1 + sin(2π logx)]dx
= E(X_{1}^{n}) + \frac{1}{\sqrt{2π}} ∫_{0}^{∞} x^{n−1} e^{−(logx)^{2}/2} sin(2π logx)dx
= E(X_{1}^{n}) + \frac{1}{\sqrt{2π}} ∫_{−∞}^{∞} e^{ny}e^{−y^{2}/2} sin(2πy)dy
= E(X_{1}^{n}) + \frac{e^{n^{2}/2}}{\sqrt{2π}} ∫_{−∞}^{∞} e^{−(y−n)^{2}/2} sin(2πy)dy
= E(X_{1}^{n}) + \frac{e^{n^{2}/2}}{\sqrt{2π}} ∫_{−∞}^{∞} e^{−s^{2}/2} sin(2π(s + n))ds
= E(X_{1}^{n}) + \frac{e^{n^{2}/2}}{\sqrt{2π}} ∫_{−∞}^{∞} e^{−s^{2}/2} sin(2πs)ds
= E(X_{1}^{n})

since e^{−s^{2}/2} sin(2πs) is an odd function. This shows that X_{1} and X_{2} have the same moments of order n = 1,2,..., but they have different distributions.

UW-Madison (Statistics)  Stat 609 Lecture 7  2014  2 / 11

beamer-tu-logo

In some cases, moments determine the distributions. The mgf, if it exists, determines a distribution.

Theorem 2.3.11

Let X and Y be random variables with cdfs F_{X} and F_{Y}, respectively. a. If X and Y are bounded, then F_{X}(u) = F_{Y}(u) for all u iff

E(X^{r}) = E(Y^{r}) for all r = 1,2,... b. If mgf’s exist in a neighborhood of 0 and M_{X}(t) = M_{Y}(t) for all t,

then F_{X}(u) = F_{Y}(u) for all u.

The key idea of the proof can be explained as follows. Note that

M_{X}(t) = ∫_{−∞}^{∞} e^{tx}f_{X}(x)dx

is the Laplace transformation of f_{X}(x). From the uniqueness of the Laplace transformation, there is a one-to-one correspondence between the mgf and the pdf. We will give a proof of this result in Chapter 4 for the multivariate case, after we introduce the characteristic functions.

UW-Madison (Statistics)  Stat 609 Lecture 7  2014  3 / 11

beamer-tu-logo

In some cases, moments determine the distributions. The mgf, if it exists, determines a distribution.

Theorem 2.3.11

Let X and Y be random variables with cdfs F_{X} and F_{Y}, respectively. a. If X and Y are bounded, then F_{X}(u) = F_{Y}(u) for all u iff

E(X^{r}) = E(Y^{r}) for all r = 1,2,... b. If mgf’s exist in a neighborhood of 0 and M_{X}(t) = M_{Y}(t) for all t,

then F_{X}(u) = F_{Y}(u) for all u.

The key idea of the proof can be explained as follows. Note that

M_{X}(t) = ∫_{−∞}^{∞} e^{tx}f_{X}(x)dx

is the Laplace transformation of f_{X}(x). From the uniqueness of the Laplace transformation, there is a one-to-one correspondence between the mgf and the pdf. We will give a proof of this result in Chapter 4 for the multivariate case, after we introduce the characteristic functions.

UW-Madison (Statistics)  Stat 609 Lecture 7  2014  3 / 11

beamer-tu-logo

In some cases, moments determine the distributions. The mgf, if it exists, determines a distribution.

Theorem 2.3.11

Let X and Y be random variables with cdfs F_{X} and F_{Y}, respectively. a. If X and Y are bounded, then F_{X}(u) = F_{Y}(u) for all u iff

E(X^{r}) = E(Y^{r}) for all r = 1,2,... b. If mgf’s exist in a neighborhood of 0 and M_{X}(t) = M_{Y}(t) for all t,

then F_{X}(u) = F_{Y}(u) for all u.

The key idea of the proof can be explained as follows. Note that

M_{X}(t) = ∫_{−∞}^{∞} e^{tx}f_{X}(x)dx

is the Laplace transformation of f_{X}(x). From the uniqueness of the Laplace transformation, there is a one-to-one correspondence between the mgf and the pdf. We will give a proof of this result in Chapter 4 for the multivariate case, after we introduce the characteristic functions.

UW-Madison (Statistics)  Stat 609 Lecture 7  2014  3 / 11

beamer-tu-logo

From the power series result in the last lecture, if the mgf of X exists in a neighborhood of 0, then it has a power series expansion which is determined by moments E(X^{n}), n = 1,2,.... Therefore, knowing the mgf and knowing moments of all order are the same, but this is under the condition that the mgf exists in a neighborhood of 0. Once we establish part (b), the proof of part (a) is easy: if X and Y are bounded, then their mgf’s exist for all t and thus their cdf’s are the same iff their moments are the same for any order. The condition that the mgf exists in a neighborhood of 0 is important. There are random variables with finite moments of any order, but their mgf’s do not exist.

Example

The pdf

f_{X}(x) = \frac{1}{\sqrt{2π}x}e^{−(logx)^{2}/2} , x ≥ 0

is called the log-normal distribution or density, because if X has pdf f_{X}, then logX has a normal pdf.

UW-Madison (Statistics)  Stat 609 Lecture 7  2014  4 / 11

beamer-tu-logo

From the power series result in the last lecture, if the mgf of X exists in a neighborhood of 0, then it has a power series expansion which is determined by moments E(X^{n}), n = 1,2,.... Therefore, knowing the mgf and knowing moments of all order are the same, but this is under the condition that the mgf exists in a neighborhood of 0. Once we establish part (b), the proof of part (a) is easy: if X and Y are bounded, then their mgf’s exist for all t and thus their cdf’s are the same iff their moments are the same for any order. The condition that the mgf exists in a neighborhood of 0 is important. There are random variables with finite moments of any order, but their mgf’s do not exist.

Example

The pdf

f_{X}(x) = \frac{1}{\sqrt{2π}x}e^{−(logx)^{2}/2} , x ≥ 0

is called the log-normal distribution or density, because if X has pdf f_{X}, then logX has a normal pdf.

UW-Madison (Statistics)  Stat 609 Lecture 7  2014  4 / 11

beamer-tu-logo

In Example 2.3.10, we have shown that the log-normal distribution has finite moments of any order. For t > 0,

M_{X}(t) = ∫_{0}^{∞} \frac{e^{tx}}{\sqrt{2π}x}e^{−(logx)^{2}/2} dx = ∞

because, when t > 0,

lim_{x→∞} \frac{e^{tx}}{\sqrt{2π}x}e^{−(logx)^{2}/2} = ∞

When t < 0,

M_{X}(t) = ∫_{0}^{∞} \frac{e^{tx}}{\sqrt{2π}x}e^{−(logx)^{2}/2} dx ≤ ∫_{0}^{∞} \frac{1}{\sqrt{2π}x}e^{−(logx)^{2}/2} dx = 1

and, hence, M_{X}(t) exists for all t < 0. How do we find a distribution for which all moments exist and the mgf does not exists for any t ≠ 0? Consider the pdf

f_{Y}(x) = \{ \begin{matrix}f_{X}(x)/2 & x > 0 \\ f_{X}(−x)/2 & x < 0 \end{matrix}

beamer-tu-logo

For this pdf,

E(|Y |^{n}) = ∫_{0}^{∞} x^{n} \frac{f_{X}(x)}{2}dx +∫_{−∞}^{0} (−x)^{n} \frac{f_{X}(−x)}{2} dx = ∫_{0}^{∞} x^{n}f_{X}(x)dx = E(X^{n})

which has been derived for any n = 1,2,... On the other hand,

E(e^{tY}) = ∫_{0}^{∞} e^{tx} \frac{f_{X}(x)}{2}dx + ∫_{−∞}^{0} e^{tx} \frac{f_{X}(−x)}{2} dx
= ∫_{0}^{∞} e^{tx} \frac{f_{X}(x)}{2}dx + ∫_{0}^{∞} e^{−tx} \frac{f_{X}(x)}{2}dx

and we have shown that one of these integrals is ∞ (depending on whether t > 0 or < 0).

Theorem.

If a random variable X has finite moment a_{n} = E(X^{n}) for any n = 1,2,..., and the series

∑_{n=0}^{∞} \frac{|a_{n}||t|^{n}}{n!} < ∞ with |t| > 0

beamer-tu-logo

E(|Y |^{n}) = ∫_{0}^{∞} x^{n} \frac{f_{X}(x)}{2}dx +∫_{−∞}^{0} (−x)^{n} \frac{f_{X}(−x)}{2} dx = ∫_{0}^{∞} x^{n}f_{X}(x)dx = E(X^{n})

which has been derived for any n = 1,2,... On the other hand,

E(e^{tY}) = ∫_{0}^{∞} e^{tx} \frac{f_{X}(x)}{2}dx + ∫_{−∞}^{0} e^{tx} \frac{f_{X}(−x)}{2} dx
= ∫_{0}^{∞} e^{tx} \frac{f_{X}(x)}{2}dx + ∫_{0}^{∞} e^{−tx} \frac{f_{X}(x)}{2}dx

and we have shown that one of these integrals is ∞ (depending on whether t > 0 or < 0).

Theorem.

If a random variable X has finite moment a_{n} = E(X^{n}) for any n = 1,2,..., and the series

∑_{n=0}^{∞} \frac{|a_{n}||t|^{n}}{n!} < ∞ with |t| > 0

beamer-tu-logo

Example.

Suppose that a_{n} = n! is the nth moment of a random variable X. Since

∑_{n=0}^{∞} \frac{|a_{n}||t|^{n}}{n!} = ∑_{n=0}^{∞}|t|^{n} = \frac{1}{1 − |t|} |t| < 1

and this function is the mgf of Gamma(1,1) at |t|, we conclude that X ∼ Gamma(1,1).

Suppose that the nth moment of a random variable Y is

a_{n} = \{ \begin{matrix}n!/(n/2)! & if n is even \\ 0 & if n is odd \end{matrix}

Then

∑_{n=0}^{∞} \frac{|a_{n}||t|^{n}}{n!} = ∑ ; n is even \frac{n!(t^{2})^{n/2}}{n!(n/2)!} = ∑_{k=0}^{∞} \frac{t^{2k}}{k!} = e^{t^{2}} t ∈ R

Later, we show that this is the mgf of N(0,\sqrt{2}), hence X ∼ N(0,\sqrt{2}).

For a log-normal distributed random variable X discussed in the beginning of this lecture, E(X^{n}) = e^{n^{2}/2} and ∑ _{n=0}^{∞}e^{n^{2}/2}|t|^{n}/n! = ∞ for any |t| > 0 and, hence, the theorem is not applicable.

UW-Madison (Statistics)  Stat 609 Lecture 7  2014  7 / 11

beamer-tu-logo

In applications we often need to approximate a cdf by a sequence of cdf’s. The next theorem gives a sufficient condition for the convergence of cdf’s and moments of random variables in terms of the convergence of mgf’s.

Theorem 2.3.12

Suppose that X_{1},X_{2},... is a sequence of random variables with mgf’s M_{X_{n}}(t), and

lim_{n→∞}M_{X_{n}}(t) = M_{X}(t) < ∞ for all t in a neighborhood of 0

where M_{X}(t) is the mgf of a random variable X. Then, for all x at which F_{X}(x) is continuous,

lim_{n→∞}F_{X_{n}}(x) = F_{X}(x).

Furthermore, for any p > 0, we have

lim_{n→∞}E|X_{n}|^{p} = E|X|^{p} and lim_{n→∞}E|X_{n}− X|^{p} = 0

beamer-tu-logo

In applications we often need to approximate a cdf by a sequence of cdf’s. The next theorem gives a sufficient condition for the convergence of cdf’s and moments of random variables in terms of the convergence of mgf’s.

Theorem 2.3.12

Suppose that X_{1},X_{2},... is a sequence of random variables with mgf’s M_{X_{n}}(t), and

lim_{n→∞}M_{X_{n}}(t) = M_{X}(t) < ∞ for all t in a neighborhood of 0

where M_{X}(t) is the mgf of a random variable X. Then, for all x at which F_{X}(x) is continuous,

lim_{n→∞}F_{X_{n}}(x) = F_{X}(x).

Furthermore, for any p > 0, we have

lim_{n→∞}E|X_{n}|^{p} = E|X|^{p} and lim_{n→∞}E|X_{n}− X|^{p} = 0

beamer-tu-logo

Example 2.3.13 (Poisson approximation)

The cdf of the binomial pmf,

f_{X}(x) = \binom{n}{x} p^{x}(1 − p)^{n−x} , x = 0,1,...,n,

where n is a positive integer and 0 < p < 1, may not be easy to calculate when n is very large. It is often approximated by the cdf of the Poisson pmf,

f_{Y}(x) = \frac{e^{−λ} λ^{x}}{x!} , x = 0,1,2,...

where λ > 0 is a constant. For this purpose, we first compute the mgf’s for the binomial and Poisson distributions.

Example 2.3.9 (binomial mgf)

Using the binomial formula

∑_{x=0}^{n} \binom{n}{x} u^{x}v^{n−x} = (u + v)^{n}

beamer-tu-logo

Example 2.3.13 (Poisson approximation)

The cdf of the binomial pmf,

f_{X}(x) = \binom{n}{x} p^{x}(1 − p)^{n−x} , x = 0,1,...,n,

where n is a positive integer and 0 < p < 1, may not be easy to calculate when n is very large. It is often approximated by the cdf of the Poisson pmf,

f_{Y}(x) = \frac{e^{−λ} λ^{x}}{x!} , x = 0,1,2,...

where λ > 0 is a constant. For this purpose, we first compute the mgf’s for the binomial and Poisson distributions.

Example 2.3.9 (binomial mgf)

Using the binomial formula

∑_{x=0}^{n} \binom{n}{x} u^{x}v^{n−x} = (u + v)^{n}

beamer-tu-logo

we obtain

M_{X}(t) = E(e^{tX}) = ∑_{x=0}^{n}e^{tx}\binom{n}{x} p^{x}(1 − p)^{n−x}

= ∑_{x=0}^{n} \binom{n}{x} (pe^{t})^{x}(1 − p)^{n−x}
= (pe^{t}+ 1 − p)^{n}

Note that

E(X) = \frac{d}{dt}M_{X}(t)|_{t=0} = n(pe^{t}+ 1 − p)^{n−1} pe^{t}|_{t=0} = np
E(X^{2}) = \frac{d^{2}}{dt^{2}}M_{X}(t)|_{t=0}
= [n(n − 1)(pe^{t}+ 1 − p)^{n−1} (pe^{t})^{2} + n(pe^{t}+ 1 − p)^{n−1} pe^{t}]|_{t=0}
= n(n − 1)p^{2} + np

We got the same results previously, but the calculation here is simpler.

UW-Madison (Statistics)  Stat 609 Lecture 7  2014  10 / 11

beamer-tu-logo

Example 2.3.13 (continued)

If Y has the Poisson pmf, then

M_{Y}(t) = ∑_{x=0}^{∞}e^{tx} \frac{e^{−λ} λ^{x}}{x!} = e^{−λ} ∑_{x=0}^{∞} \frac{(e^{t}λ)^{x}}{x!} = e^{−λ} e^{e^{t}λ} = e^{λ(e^{t}−1)}

which is finite for any t ∈ R. Let X_{n} have the binomial distribution with n and p. Suppose that lim_{n→∞} np = λ > 0 (that means p also depends on n and p → 0 when n → ∞). Then, for any t, as n → ∞,

M_{X_{n}}(t) = (pe^{t}+ 1 − p)^{n} = [1 + \frac{(np)(e^{t}− 1)}{n} ]^{n} → e^{λ(e^{t}−1)} = M_{Y}(t)

using the fact that, for any sequence of numbers a_{n} converges to a,

lim_{n→∞}( 1 + \frac{a_{n}}{n} ) ^{n} = e^{a}

With this result and Theorem 2.3.12, we can approximate P(X_{n} ≤ u) by P(Y ≤ u) when n is large and np is approximately a constant.

UW-Madison (Statistics)  Stat 609 Lecture 7  2014  11 / 11