beamer-tu-logo

Lecture 21: Multivariate mgf’s and chf’s

Multivariate mgf and chf

For an n-dimensional random vector X, its mgf is defined as

M_{X}(t) = E(e^{t′X}), t ∈ R^{n}

and its chf is defined as

ϕ_{X}(t) = E(e^{i̇t′X}), t ∈ R^{n}

Simple properties of mgf’s and chf’s

M_{X}(t) is either finite or ∞, and M_{X}(0) = 1. The chf ϕ_{X}(t) is a continuous function of t ∈ R^{n} and ϕ_{X}(0) = 1. For any fixed k × n matrix A and vector b ∈ R^{k}, the mgf and chf of the random vector AX + b are, respectively, M_{AX+b}(t) = e^{b′t}M_{X}(A′t) and ϕ_{AX+b}(t) = e^{i̇b′t}ϕ_{X}(A′t), t ∈ R^{k} In particular, if Y is the first m components of X, m < n, then the mgf and chf of Y are, respectively, M_{Y}(s) = M_{X}((s,0)) and ϕ_{Y}(s) = ϕ_{X}((s,0)), s ∈ R^{m}, 0 ∈ R^{n−m}

UW-Madison (Statistics)  Stat 609 Lecture 21  2014  1 / 11

beamer-tu-logo

Lecture 21: Multivariate mgf’s and chf’s

Multivariate mgf and chf

For an n-dimensional random vector X, its mgf is defined as

M_{X}(t) = E(e^{t′X}), t ∈ R^{n}

and its chf is defined as

ϕ_{X}(t) = E(e^{i̇t′X}), t ∈ R^{n}

Simple properties of mgf’s and chf’s

M_{X}(t) is either finite or ∞, and M_{X}(0) = 1. The chf ϕ_{X}(t) is a continuous function of t ∈ R^{n} and ϕ_{X}(0) = 1. For any fixed k × n matrix A and vector b ∈ R^{k}, the mgf and chf of the random vector AX + b are, respectively, M_{AX+b}(t) = e^{b′t}M_{X}(A′t) and ϕ_{AX+b}(t) = e^{i̇b′t}ϕ_{X}(A′t), t ∈ R^{k} In particular, if Y is the first m components of X, m < n, then the mgf and chf of Y are, respectively, M_{Y}(s) = M_{X}((s,0)) and ϕ_{Y}(s) = ϕ_{X}((s,0)), s ∈ R^{m}, 0 ∈ R^{n−m}

UW-Madison (Statistics)  Stat 609 Lecture 21  2014  1 / 11

beamer-tu-logo

Calculation of moments

If X = (X_{1},...,X_{n}) and M_{X}(t) is finite in a neighborhood of 0, then E(X_{1}^{r_{1}}··· X_{n}^{r_{n}}) is finite for any nonnegative integers r_{1},...,r_{n}, and

E(X_{1}^{r_{1}}··· X_{n}^{r_{n}}) = \frac{∂^{r_{1}+··· +r_{n}}M_{X}(t)}{∂t_{1}^{r_{1}}··· ∂t_{n}^{r_{n}}}|_{t=0}

In particular,

\frac{∂M_{X}(t)}{∂t}|_{t=0} = E(X) = ( E(X_{1}) ⋮ E(X_{n}) )

and

\frac{∂^{2}M_{X}(t)}{∂t∂t′}|_{t=0} = E(XX′) = ( \begin{matrix}E(X_{1}^{2}) & E(X_{1}X_{2}) & ··· & E(X_{1}X_{n}) \\ E(X_{2}X_{1}) & E(X_{2}^{2}) & ··· & E(X_{2}X_{n}) \\ ··· & ··· & ··· & ··· \\ E(X_{n}X_{1}) & E(X_{n}X_{2}) & ··· & E(X_{n}^{2}) \end{matrix} )

When the mgf is not finite in any neighborhood of 0, then we can use the chf to calculate moments.

UW-Madison (Statistics)  Stat 609 Lecture 21  2014  2 / 11

beamer-tu-logo

Suppose that E|X_{1}^{r_{1}}··· X_{k}^{r_{n}}| < ∞ for some nonnegative integers r_{1},...,r_{n} with r = r_{1} + ··· + r_{n}. Since

| \frac{∂^{r}e^{i̇t′X}}{∂t_{1}^{r_{1}}··· ∂t_{k}^{r_{n}}}| = |i̇^{r}X_{1}^{r_{1}}··· X_{k}^{r_{n}}e^{i̇t′X}| ≤ |X_{1}^{r_{1}}··· X_{k}^{r_{n}}|

which is integrable, we can switch integration and differentiation to get

\frac{∂^{r}ϕ_{X}(t)}{∂t_{1}^{r_{1}}··· ∂t_{k}^{r_{n}}}= i̇^{r}E ( X_{1}^{r_{1}}··· X_{k}^{r_{n}}e^{i̇t′X})

and

\frac{∂^{r}ϕ_{X}(t)}{∂t_{1}^{r_{1}}··· ∂t_{k}^{r_{n}}}|_{t=0} = i̇^{r}E(X_{1}^{r_{1}}··· X_{k}^{r_{n}}).

In particular,

\frac{∂ϕ_{X}(t)}{∂t}|_{t=0} = i̇EX, \frac{∂^{2}ϕ_{X}(t)}{∂t∂t^{τ}}|_{t=0} = −E(XX′)

Theorem M1.

If the mgf M_{X}(t) of an n-dimensional random vector X is finite in a neighborhood of 0, then the chf of X is ϕ_{X}(t) = M_{X}(i̇t), t ∈ R^{n}.

UW-Madison (Statistics)  Stat 609 Lecture 21  2014  3 / 11

beamer-tu-logo

Suppose that E|X_{1}^{r_{1}}··· X_{k}^{r_{n}}| < ∞ for some nonnegative integers r_{1},...,r_{n} with r = r_{1} + ··· + r_{n}. Since

| \frac{∂^{r}e^{i̇t′X}}{∂t_{1}^{r_{1}}··· ∂t_{k}^{r_{n}}}| = |i̇^{r}X_{1}^{r_{1}}··· X_{k}^{r_{n}}e^{i̇t′X}| ≤ |X_{1}^{r_{1}}··· X_{k}^{r_{n}}|

which is integrable, we can switch integration and differentiation to get

\frac{∂^{r}ϕ_{X}(t)}{∂t_{1}^{r_{1}}··· ∂t_{k}^{r_{n}}}= i̇^{r}E ( X_{1}^{r_{1}}··· X_{k}^{r_{n}}e^{i̇t′X})

and

\frac{∂^{r}ϕ_{X}(t)}{∂t_{1}^{r_{1}}··· ∂t_{k}^{r_{n}}}|_{t=0} = i̇^{r}E(X_{1}^{r_{1}}··· X_{k}^{r_{n}}).

In particular,

\frac{∂ϕ_{X}(t)}{∂t}|_{t=0} = i̇EX, \frac{∂^{2}ϕ_{X}(t)}{∂t∂t^{τ}}|_{t=0} = −E(XX′)

Theorem M1.

If the mgf M_{X}(t) of an n-dimensional random vector X is finite in a neighborhood of 0, then the chf of X is ϕ_{X}(t) = M_{X}(i̇t), t ∈ R^{n}.

UW-Madison (Statistics)  Stat 609 Lecture 21  2014  3 / 11

beamer-tu-logo

Proof.

We first prove this for a univariate X. If M_{X}(t) is finite in a neighborhood of 0, (−δ,δ), 0 < δ < 1, then M_{X}(t) is differentiable of all order in (−δ,δ), E|X|^{r} < ∞ for all r = 1,2..., and

M_{X}(t) = ∑_{k=0}^{∞} \frac{E(X^{k})t^{k}}{k!}, t ∈ (−δ,δ)

Using the inequality

|e^{i̇sx} [e^{i̇tx}− ∑_{k=0}^{n} \frac{(i̇tx)^{k}}{k!}] | ≤ \frac{|tx|^{n+1}}{(n + 1)!} t,s ∈ R

we obtain that

|ϕ_{X}(s + t) − ∑_{k=0}^{n} \frac{i̇^{k}t^{k}}{k!}E(X^{k}e^{i̇sX})| ≤ \frac{|t|^{n+1}E|X|^{n+1}}{(n + 1)!}, |t| < δ, s ∈ R

which implies that

ϕ_{X}(s + t) = ∑_{k=0}^{∞} \frac{ϕ_{X}^{(k)}(s)t^{k}}{k!} |t| < δ, s ∈ R

where g^{(k)}(s) = d^{k}g(s)/ds^{k}.

UW-Madison (Statistics)  Stat 609 Lecture 21  2014  4 / 11

beamer-tu-logo

Setting s = 0, we obtain that

ϕ_{X}(t) = ∑_{k=0}^{∞} \frac{ϕ_{X}^{(k)}(0)t^{k}}{k!} = ∑_{k=0}^{∞} \frac{i̇^{k}E(X^{k})t^{k}}{k!}= ∑_{k=0}^{∞} \frac{E(X^{k})(i̇t)^{k}}{k!}= M_{X}(i̇t) |t| < δ

For |t| < δ and |s| < δ,

ϕ_{X}(s + t) = ∑_{k=0}^{∞} \frac{ϕ_{X}^{(k)}(s)t^{k}}{k!} = ∑_{k=0}^{∞} \frac{t^{k}}{k!} ∑_{j=k}^{∞} \frac{i̇^{j}E(X^{j})s^{j−k}}{(j − k)!}
= ∑_{j=0}^{∞} ∑_{k=0}^{j}i̇^{j}E(X^{j})\frac{t^{k}s^{j−k}}{k!(j − k)!} = ∑_{j=0}^{∞} ∑_{k=0}^{j} \frac{i̇^{j}E(X^{j})(s + t)^{j}}{j!}

This proves that

ϕ_{X}(s) = M_{X}(i̇s), |s| < 2δ

Continuing this process, we can show that ϕ_{X}(s) = M_{X}(i̇s) for |s| < 3δ, |s| < 4δ,..., and hence, ϕ_{X}(t) = M_{X}(i̇t) for t ∈ R. Consider now an n-dimensional X. From the proof result for the univariate case, we have

ϕ_{X}(t) = E(e^{i̇t′X}) = ϕ_{t′X}(1) = M_{t′X}(i̇) = M_{X}(i̇t), t ∈ R^{n}.

UW-Madison (Statistics)  Stat 609 Lecture 21  2014  5 / 11

beamer-tu-logo

Theorem M2 (uniqueness)

(i) Random vectors X ∼ Y iff the chf’s ϕ_{X}(t) = ϕ_{Y}(t) for t ∈ R^{n}. (ii) If the mgf’s of random vectors X and Y satisfy M_{X}(t) = M_{Y}(t) < ∞

for t in a neighborhood of 0, then X ∼ Y .

Proof.

It is clear that the distribution of X determines its mgf and chf. Part (i) follows from the following multivariate inversion formula: for any A = (a_{1},b_{1}) × ··· × (a_{n},b_{n}) such that the distribution of X is continuous at all points in the boundary of A,

P(X ∈ A) = lim_{c→∞}∫_{−c}^{c} ··· ∫_{−c}^{c} \frac{ϕ_{X}(t_{1},...,t_{n})}{i̇^{n}(2π)^{n}}∏_{i=1}^{n} \frac{e^{−i̇t_{i}a_{i}} − e^{−i̇t_{i}b_{i}}}{t_{i}} dt_{i}.

To establish (ii), we use the relationship between mgf and chf shown in Theorem M1. If the mgf’s M_{X}(t) = M_{Y}(t) < ∞ for t in a neighborhood of 0, then

ϕ_{X}(t) = M_{X}(i̇t) = M_{Y}(i̇t) = ϕ_{Y}(t), t ∈ R^{n}

Then by (i), X and Y have the same distribution.

UW-Madison (Statistics)  Stat 609 Lecture 21  2014  6 / 11

beamer-tu-logo

Similar to the univariate case, convergence of chf’s and cdf’s are equivalent and convergence of mgf’s implies convergence of cdf’s. We introduce the following result without giving proofs.

Theorem M3.

Suppose that X_{1},X_{2},... is a sequence of k-dimensional random vectors with mgf’s M_{X_{n}}(t) and chf’s ϕ_{X_{n}}(t), t ∈ R^{k}.

(i) If lim_{n→∞} M_{X_{n}}(t) = M_{X}(t) < ∞ for all t in a neighborhood of 0, where M_{X}(t) is the mgf of a k-dimensional random vector X, then lim_{n→∞} F_{X_{n}}(x) = F_{X}(x) for all x where F_{X}(x) is continuous.

(ii) A necessary and sufficient condition for the convergence of F_{X_{n}}(x)’s in (i) is that lim_{n→∞} ϕ_{X_{n}}(t) = ϕ_{X}(t) for all t ∈ R^{k}, where ϕ_{X}(t) is the chf of a k-dimensional random vector X.

(iii) In (ii), if lim_{n→∞} ϕ_{X_{n}}(t) = g(t) for all t ∈ R^{k} and g(t) is a continuous function on R^{k}, then g(t) must be a chf of some k-dimensional random vector X and the result in (ii) holds.

Note that Theorem M3(ii) gives a necessary and sufficient condition, while the converse of Theorem M3(i) is not true in general.

UW-Madison (Statistics)  Stat 609 Lecture 21  2014  7 / 11

beamer-tu-logo

The next result concerns the relationship between independence and chf’s and mgf’s.

Theorem M4.

Let X and Y be two random vectors with dimensions n and m, respectively.

(i) X and Y are independent iff their joint and marginal chf’s satisfy ϕ_{(X,Y )}(t) = ϕ_{X}(t_{1})ϕ_{Y}(t_{2}) for all t = (t_{1},t_{2}), t_{1} ∈ R^{n}, t_{2} ∈ R^{m}.

(ii) If the mgf of (X,Y ) is finite in a neighborhood of 0 ∈ R^{n+m}, then X and Y are independent iff their joint and marginal mgf’s satisfy M_{(X,Y )}(t)=M_{X}(t_{1})M_{Y}(t_{2}) for all t =(t_{1},t_{2}) in the neighborhood of 0.

Proof.

Because of Theorem M1, (ii) follows from (i). The only if part of (i) follows from Theorem 4.2.12A. It remains to show the if part of (i). Let F(x,y) be the cdf of (X,Y ), F_{X}(x) be the cdf of X, and F_{Y}(y) be the cdf of Y , x ∈ R^{n}, y ∈ R^{m}.

UW-Madison (Statistics)  Stat 609 Lecture 21  2014  8 / 11

beamer-tu-logo

Define

F̃(x,y) = F_{X}(x)F_{Y}(y), x ∈ R^{n}, y ∈ R^{m}

It can be easily shown that F̃ is a cdf on R^{n+m}. There exist random vectors X̃ (n-dimensional) and Ỹ (m-dimensional) such that the cdf of (X̃,Ỹ) is F̃(x,y). From the form of F̃, we know that X̃ and Ỹ are independent, X̃ ∼ F_{X} with chf ϕ_{X}, and Ỹ ∼ F_{Y} with chf ϕ_{Y}. Hence the chf of F̃ must be ϕ_{X}(t_{1})ϕ_{Y}(t_{2}), t_{1} ∈ R^{n}, t_{2} ∈ R^{m}. If ϕ_{(X,Y )}(t) = ϕ_{X}(t_{1})ϕ_{Y}(t_{2}) for all t = (t_{1},t_{2}), t_{1} ∈ R^{n}, t_{2} ∈ R^{m}, then, by uniqueness, F(x,y) = F̃(x,y) = F_{X}(x)F_{Y}(y) for all x ∈ R^{n}, y ∈ R^{m}. This proves that X and Y are independent.

As an example, we now consider the mgf’s in a family of multivariate distributions that is an extension of the univariate normal distribution family.

n-dimensional multivariate normal distribution

Let μ ∈ R^{n}, Σ be a positive definite n × n matrix, and |Σ| be the determinant of Σ.

UW-Madison (Statistics)  Stat 609 Lecture 21  2014  9 / 11

beamer-tu-logo

Define

F̃(x,y) = F_{X}(x)F_{Y}(y), x ∈ R^{n}, y ∈ R^{m}

It can be easily shown that F̃ is a cdf on R^{n+m}. There exist random vectors X̃ (n-dimensional) and Ỹ (m-dimensional) such that the cdf of (X̃,Ỹ) is F̃(x,y). From the form of F̃, we know that X̃ and Ỹ are independent, X̃ ∼ F_{X} with chf ϕ_{X}, and Ỹ ∼ F_{Y} with chf ϕ_{Y}. Hence the chf of F̃ must be ϕ_{X}(t_{1})ϕ_{Y}(t_{2}), t_{1} ∈ R^{n}, t_{2} ∈ R^{m}. If ϕ_{(X,Y )}(t) = ϕ_{X}(t_{1})ϕ_{Y}(t_{2}) for all t = (t_{1},t_{2}), t_{1} ∈ R^{n}, t_{2} ∈ R^{m}, then, by uniqueness, F(x,y) = F̃(x,y) = F_{X}(x)F_{Y}(y) for all x ∈ R^{n}, y ∈ R^{m}. This proves that X and Y are independent.

As an example, we now consider the mgf’s in a family of multivariate distributions that is an extension of the univariate normal distribution family.

n-dimensional multivariate normal distribution

Let μ ∈ R^{n}, Σ be a positive definite n × n matrix, and |Σ| be the determinant of Σ.

UW-Madison (Statistics)  Stat 609 Lecture 21  2014  9 / 11

beamer-tu-logo

The following pdf on R^{n},

f (x) = \frac{1}{(2π)^{n/2}|Σ|^{1/2}} exp( −\frac{(x − μ)′Σ^{−1} (x − μ)}{2} ) , x ∈ R^{n},

is called the n-dimensional normal pdf and the corresponding distribution is called the n-dimensional normal distribution and denoted by N(μ,Σ).

Using transformation, we can show that f (x) is indeed a pdf on R^{n}. If X ∼ N(0,Σ), then, for any t ∈ R^{n},

M_{X}(t)= \frac{1}{(2π)^{n/2}|Σ|^{1/2}} ∫_{R^{n}} e^{t′x}exp( −\frac{x′Σ^{−1} x}{2}) dx
= \frac{e^{t′Σt/2}}{(2π)^{n/2}|Σ|^{1/2}} ∫_{R^{n}} exp( −\frac{(x − Σt)′Σ^{−1} (x − Σt)}{2} ) dx
= e^{t′Σt/2}

If X ∼ N(μ,Σ), an application of the property of mgf and the previous result leads to

M_{X}(t) = e^{μ′t+t′Σt/2} t ∈ R^{n}
UW-Madison (Statistics)  Stat 609 Lecture 21  2014  10 / 11

beamer-tu-logo

From the property of mgf, we can differentiate M_{X}(t) to get

E(X) = \frac{∂M_{X}(t)}{∂t}|_{t=0} = (μ + Σt)e^{μ′t+t′Σt/2}|_{t=0} = μ

E(XX′)= \frac{∂^{2}M_{X}(t)}{∂t∂t′}|_{t=0} = \frac{∂}{∂t}(μ + Σt)e^{μ′t+t′Σt/2}|_{t=0}
= [Σe^{μ′t+t′Σt/2}+ (μ + Σt)(μ + Σt)′e^{μ′t+t′Σt/2}] |_{t=0}
= Σ + μμ′

Therefore, Var(X) = E(XX′) − E(X)E(X′) = Σ

Like the univariate case, μ and Σ in N(μ,Σ) are respectively the mean vector and covariance matrix of the distribution. Another important consequence from the previous result is that the components of X ∼ N(μ,Σ) are independent iff Σ is a diagonal matrix, i.e., components of X are uncorrelated.

UW-Madison (Statistics)  Stat 609 Lecture 21  2014  11 / 11