beamer-tu-logo

Lecture 22: Multivariate normal distributions (part I)

We now study properties of the multivariate normal distribution defined in the last lecture.

Bivariate normal distributions

First, we want to show that the definition for the general multivariate normal distribution with n = 2 is consistent with the early definition of the bivariate normal distribution. Let μ_{1} ∈ R, μ_{2} ∈ R, σ_{1} > 0, σ_{2} > 0, and −1 < ρ < 1 be constants, and

μ = ( \begin{matrix}μ_{1} \\ μ_{2} \end{matrix} )  Σ = ( \begin{matrix}σ_{1}^{2} & ρσ_{1}σ_{2} \\ ρσ_{1}σ_{2} & σ_{2}^{2} \end{matrix} )  x = ( \begin{matrix}x_{1} \\ x_{2} \end{matrix} )

Then |Σ| = σ_{1}^{2}σ_{2}^{2} − ρ^{2}σ_{1}^{2}σ_{2}^{2} = σ_{1}^{2}σ_{2}^{2}(1 − ρ^{2})

Σ^{−1} = \frac{1}{σ_{1}^{2}σ_{2}^{2}(1 − ρ^{2})} ( \begin{matrix}σ_{2}^{2} & −ρσ_{1}σ_{2} \\ −ρσ_{1}σ_{2} & σ_{1}^{2} \end{matrix} )

(x −μ)′Σ^{−1} (x −μ) = \frac{1}{|Σ|}( \begin{matrix}x_{1} − μ_{1} \\ x_{2} − μ_{2} \end{matrix} ) ′( \begin{matrix}σ_{2}^{2} & −ρσ_{1}σ_{2} \\ −ρσ_{1}σ_{2} & σ_{1}^{2} \end{matrix} )( \begin{matrix}x_{1} − μ_{1} \\ x_{2} − μ_{2} \end{matrix} )

UW-Madison (Statistics)  Stat 609 Lecture 22  2014  1 / 12

beamer-tu-logo

= \frac{1}{|Σ|}( \begin{matrix}x_{1} − μ_{1} \\ x_{2} − μ_{2} \end{matrix} ) ′( \begin{matrix}σ_{2}^{2}(x_{1} − μ_{1}) − ρσ_{1}σ_{2}(x_{2} − μ_{2}) \\ −ρσ_{1}σ_{2}(x_{1} − μ_{1}) + σ_{1}^{2}(x_{2} − μ_{2}) \end{matrix} )
= \frac{σ_{2}^{2}(x_{1} − μ_{1})^{2} − 2ρσ_{1}σ_{2}(x_{1} − μ_{1})(x_{2} − μ_{2}) + σ_{1}^{2}(x_{2} − μ_{2})^{2}}{σ_{1}^{2}σ_{2}^{2}(1 − ρ^{2})}
= \frac{(x_{1} − μ_{1})^{2}}{σ_{1}^{2}(1 − ρ^{2})} − \frac{2ρ(x_{1} − μ_{1})(x_{2} − μ_{2})}{σ_{1}σ_{2}(1 − ρ^{2})} + \frac{(x_{2} − μ_{2})^{2}}{σ_{2}^{2}(1 − ρ^{2})}

Thus, when n = 2,

f (x) = \frac{1}{(2π)^{n/2}|Σ|^{1/2}} exp( −\frac{(x − μ)′Σ^{−1} (x − μ)}{2} )

= \frac{exp( −\frac{(x_{1}−μ_{1})^{2}}{2σ_{1}^{2}(1−ρ^{2})} + \frac{ρ(x_{1}−μ_{1})(x_{2}−μ_{2})}{σ_{1}σ_{2}(1−ρ^{2})} − \frac{(x_{2}−μ_{2})^{2}}{2σ_{2}^{2}(1−ρ^{2})} )}{2πσ_{1}σ_{2}\sqrt{1 − ρ^{2}}}

which is the same as the bivariate normal pdf defined earlier.

Transformation and marginals

If X is an n-dimensional random vector ∼ N(μ,Σ) and Y = AX + b, where A is a fixed k × n matrix of rank k ≤ n and b is a fixed k-dimensional vector, then Y ∼ N(Aμ,AΣA′).

UW-Madison (Statistics)  Stat 609 Lecture 22  2014  2 / 12

beamer-tu-logo

= \frac{1}{|Σ|}( \begin{matrix}x_{1} − μ_{1} \\ x_{2} − μ_{2} \end{matrix} ) ′( \begin{matrix}σ_{2}^{2}(x_{1} − μ_{1}) − ρσ_{1}σ_{2}(x_{2} − μ_{2}) \\ −ρσ_{1}σ_{2}(x_{1} − μ_{1}) + σ_{1}^{2}(x_{2} − μ_{2}) \end{matrix} )
= \frac{σ_{2}^{2}(x_{1} − μ_{1})^{2} − 2ρσ_{1}σ_{2}(x_{1} − μ_{1})(x_{2} − μ_{2}) + σ_{1}^{2}(x_{2} − μ_{2})^{2}}{σ_{1}^{2}σ_{2}^{2}(1 − ρ^{2})}
= \frac{(x_{1} − μ_{1})^{2}}{σ_{1}^{2}(1 − ρ^{2})} − \frac{2ρ(x_{1} − μ_{1})(x_{2} − μ_{2})}{σ_{1}σ_{2}(1 − ρ^{2})} + \frac{(x_{2} − μ_{2})^{2}}{σ_{2}^{2}(1 − ρ^{2})}

Thus, when n = 2,

f (x) = \frac{1}{(2π)^{n/2}|Σ|^{1/2}} exp( −\frac{(x − μ)′Σ^{−1} (x − μ)}{2} )

= \frac{exp( −\frac{(x_{1}−μ_{1})^{2}}{2σ_{1}^{2}(1−ρ^{2})} + \frac{ρ(x_{1}−μ_{1})(x_{2}−μ_{2})}{σ_{1}σ_{2}(1−ρ^{2})} − \frac{(x_{2}−μ_{2})^{2}}{2σ_{2}^{2}(1−ρ^{2})} )}{2πσ_{1}σ_{2}\sqrt{1 − ρ^{2}}}

which is the same as the bivariate normal pdf defined earlier.

Transformation and marginals

If X is an n-dimensional random vector ∼ N(μ,Σ) and Y = AX + b, where A is a fixed k × n matrix of rank k ≤ n and b is a fixed k-dimensional vector, then Y ∼ N(Aμ,AΣA′).

UW-Madison (Statistics)  Stat 609 Lecture 22  2014  2 / 12

beamer-tu-logo

This can be proved by using multivariate transformation, or the mgf. In the last lecture we showed that the mgf of X ∼ N(μ,Σ) is

M_{X}(t) = e^{μ′t+t′Σt/2} t ∈ R^{n}

From the properties of mgf, Y = AX + b has mgf

M_{Y}(t) = e^{b′t}M_{X}(A′t) = e^{b′t}e^{μ′(A′t)+(A′t)′Σ(A′t)/2}= e^{(Aμ)′t+t′(AΣA′)t/2}, t ∈ R^{k}

which is exactly the mgf for N(Aμ,AΣA′), noting that (AΣA′)^{−1} exists since A has rank k ≤ n and Σ has rank n. The result seems still hold even if AΣA′is singular, but we have not defined the normal distribution with a singular covariance matrix. If we take A = (I_{k} 0), where I_{k} is the identity matrix of order k and 0 is the k × (n − k) matrix of all 0’s, then AX is exactly the vector containing the first k components of X and, therefore, we have shown that

If Y is multivariate normal, then any sub-vector of Y is also normally distributed.

Is the converse true? That is, if both X and Y are normal, should the joint pdf of (X,Y ) be always normal?

UW-Madison (Statistics)  Stat 609 Lecture 22  2014  3 / 12

beamer-tu-logo

A counter-example

Consider

f (x,y) = \frac{1}{2π} e^{−(x^{2}+y^{2})/2} [1 + g(x,y)], (x,y) ∈ R^{2}

where

g(x,y) = \{ \begin{matrix}xy & −1 < x < 1, −1 < y < 1 \\ 0 & otherwise \end{matrix}

Then f (x,y) ≥ 0. (Is it a pdf?)

∫_{−∞}^{∞} f (x,y)dy = \frac{1}{2π} [∫_{−∞}^{∞} e^{−(x^{2}+y^{2})/2} dy + ∫_{−∞}^{∞} e^{−(x^{2}+y^{2})/2} g(x,y)dy]
= \frac{1}{2π} e^{−x^{2}/2} [∫_{−∞}^{∞} e^{−y^{2}/2} dy + xI(|x| < 1)∫_{−1}^{1} ye^{−y^{2}/2} dy]
= \frac{1}{\sqrt{2π}} e^{−x^{2}/2}

i.e., the marginal of X is N(0,1) (which also shows that f (x,y) is a pdf). Similarly, the pdf of Y is N(0,1). However, this f (x,y) is certainly not a bivariate normal pdf.

UW-Madison (Statistics)  Stat 609 Lecture 22  2014  4 / 12

beamer-tu-logo

Other joint distributions may not have the property that the marginal distributions are of the same type as the joint distribution.

Uniform distributions

A general n-dimensional uniform distribution on a set A ⊂ R^{n} has pdf

f (x) = \{ \begin{matrix}1/C & x ∈ A \\ 0 & x ∉ A \end{matrix} C = ∫_{A}dx

For n = 1 and A = an interval, C is the length of the interval. For n = 2, C is the area of the set A. Consider for example n = 2 and A is the rectangle

A = [a,b] × [c,d] = {(x,y) ∈ R^{2} : a ≤ x ≤ b, c ≤ y ≤ d}

where a, b, c and d are constants. In this case, the two marginal distributions are uniform distributions on intervals [a,b] and [c,d], since, when x ∈ [a,b],

∫_{−∞}^{∞} f (x,y)dy = \frac{1}{C} ∫_{c}^{d} dy = \frac{1}{(b − a)(d − c)} ∫_{c}^{d} dy = \frac{1}{b − a}

and ∫_{−∞}^{∞} f (x,y)dy = 0 when x ∉ [a,b].

UW-Madison (Statistics)  Stat 609 Lecture 22  2014  5 / 12

beamer-tu-logo

However, if A is a disk

A = {(x,y) ∈ R^{2} : x^{2} + y^{2} ≤ 1}

then C = π and, when x ∈ [−1,1],

∫_{−∞}^{∞} f (x,y)dy = ∫_{−\sqrt{1−x^{2}}}^{√ \overline{1−x^{2}}} \frac{1}{π} dy = \frac{2\sqrt{1 − x^{2}}}{π}

and ∫_{−∞}^{∞} f (x,y)dy = 0 when x ∉ [−1,1]. This shows that the marginal distributions are not uniform.

beamer-tu-logo

Normal distributions with singular covariance matrices

Consider an n-dimensional X ∼ N(μ,Σ) with a positive definite Σ and a fixed k × n matrix A that is not of rank k (so k may be larger than n). The mgf of Y = AX is still equal to

M_{Y}(t) = e^{(Aμ)′t+t′(AΣA′)t/2}, t ∈ R^{k}

But what is the distribution corresponding to this mgf?

Lemma.

For any n × n non-negative definite matrix Σ and μ ∈ R^{n}, the function e^{μ′t+t′Σt/2}defined for all t ∈ R^{n} is the mgf of an n-dimensional random vector X.

Proof.

From the theory of linear algebra, a non-negative definite matrix Σ of rank r < n satisfies

Σ = T′( \begin{matrix}Λ & 0 \\ 0 & 0 \end{matrix} ) T = C′ΛC T = ( \begin{matrix}C \\ D \end{matrix} )

beamer-tu-logo

Normal distributions with singular covariance matrices

Consider an n-dimensional X ∼ N(μ,Σ) with a positive definite Σ and a fixed k × n matrix A that is not of rank k (so k may be larger than n). The mgf of Y = AX is still equal to

M_{Y}(t) = e^{(Aμ)′t+t′(AΣA′)t/2}, t ∈ R^{k}

But what is the distribution corresponding to this mgf?

Lemma.

For any n × n non-negative definite matrix Σ and μ ∈ R^{n}, the function e^{μ′t+t′Σt/2}defined for all t ∈ R^{n} is the mgf of an n-dimensional random vector X.

Proof.

From the theory of linear algebra, a non-negative definite matrix Σ of rank r < n satisfies

Σ = T′( \begin{matrix}Λ & 0 \\ 0 & 0 \end{matrix} ) T = C′ΛC T = ( \begin{matrix}C \\ D \end{matrix} )

beamer-tu-logo

positive, 0 denotes a matrix of 0’s of an appropriate order, C is an r × n matrix of rank r, T is an n × n matrix satisfying TT′= T′T = I_{n} (the identity matrix of order n), CC′= I_{r}, DC′= 0, DD′= I_{n−r} , and C′C + D′D = I_{n}. Let Y be an r-dimensional random vector ∼ N(Cμ,Λ) and define

X = T′( \begin{matrix}Y \\ Dμ \end{matrix} ) = C′Y + D′Dμ

Since Y ∼ N(Cμ,Λ), its mgf is M_{Y}(s) = e^{(Cμ)′s+s′Λs/2} , s ∈ R^{r} and the mgf of X is

M_{X}(t)= e^{(D′Dμ)′t}M_{Y}(Ct)) = e^{(D′Dμ)′t}e^{(Cμ)′(Ct)+(Ct)′Λ(Ct)/2}
= e^{μ′(D′D+C′C)t+t′C′ΛCt/2}= e^{μ′t+t′Σt/2} t ∈ R^{n}

This completes the proof.

Definition

For any fixed n × n non-negative definite matrix Σ and μ ∈ R^{n}, the distribution of an n-dimensional random vector with mgf e^{μ′t+t′Σt/2}is called normal distribution and denoted by N(μ,Σ).

UW-Madison (Statistics)  Stat 609 Lecture 22  2014  8 / 12

beamer-tu-logo

positive, 0 denotes a matrix of 0’s of an appropriate order, C is an r × n matrix of rank r, T is an n × n matrix satisfying TT′= T′T = I_{n} (the identity matrix of order n), CC′= I_{r}, DC′= 0, DD′= I_{n−r} , and C′C + D′D = I_{n}. Let Y be an r-dimensional random vector ∼ N(Cμ,Λ) and define

X = T′( \begin{matrix}Y \\ Dμ \end{matrix} ) = C′Y + D′Dμ

Since Y ∼ N(Cμ,Λ), its mgf is M_{Y}(s) = e^{(Cμ)′s+s′Λs/2} , s ∈ R^{r} and the mgf of X is

M_{X}(t)= e^{(D′Dμ)′t}M_{Y}(Ct)) = e^{(D′Dμ)′t}e^{(Cμ)′(Ct)+(Ct)′Λ(Ct)/2}
= e^{μ′(D′D+C′C)t+t′C′ΛCt/2}= e^{μ′t+t′Σt/2} t ∈ R^{n}

This completes the proof.

Definition

For any fixed n × n non-negative definite matrix Σ and μ ∈ R^{n}, the distribution of an n-dimensional random vector with mgf e^{μ′t+t′Σt/2}is called normal distribution and denoted by N(μ,Σ).

UW-Madison (Statistics)  Stat 609 Lecture 22  2014  8 / 12

beamer-tu-logo

If Σ is positive definition, then this definition is the same as the previous definition using the pdf. If X ∼ N(μ,Σ) and Y = AX + b, then Y ∼ N(Aμ,AΣA′), regardless of whether AΣA′is singular or not. If X is multivariate normal, then any sub-vector of X is also normally distributed. If n-dimensional X ∼ N(μ,Σ) and the rank of Σ is r < n, there exists an r × n matrix C of rank r and Y = CX ∼ N(Cμ,CΣC′), where CΣC′is a diagonal matrix whose diagonal elements are all positive, and hence Y has an r-dimensional normal pdf and components of Y are independent. If n-dimensional X ∼ N(μ,Σ) and the rank of Σ is r < n, then, from the previous discussion, X = C′Y + D′Dμ, where Y ∼ N(Cμ,CΣC′) and

E(X) = C′E(Y ) + D′Dμ = (C′C + D′D)μ = μ
Var(X) = C′Var(Y )C = C′CΣC′C = Σ

beamer-tu-logo

Thus, μ and Σ in N(μ,Σ) is still the mean and covariance matrix. Furthermore, any two components of X ∼ N(μ,Σ) are independent iff they are uncorrelated. This can be shown as follows. Suppose that X_{1} and X_{2} are the first two components of X and Cov(X_{1},X_{2}) = 0, i.e., the (1,2)th and (2,1)th elements of Σ are 0. Let μ_{1} and μ_{2} be the first two components of μ and σ_{1}^{2} and σ_{2}^{2} be the first and second diagonal elements of Σ, and let t = (t_{1},t_{2},0,...,0), t_{1} ∈ R, t_{2} ∈ R. Then the mgf of (X_{1},X_{2}) is

M_{(X_{1},X_{2})}(t_{1},t_{2}) = e^{μ′t+t′Σt/2}= e^{μ_{1}t_{1}+σ_{1}^{2}t_{1}^{2}/2}e^{μ_{2}t_{2}+σ_{2}^{2}t_{2}^{2}/2} t_{1} ∈ R, t_{2} ∈ R

By Theorem M4, X_{1} and X_{2} are independent.

Theorem.

An n-dimensional random vector X ∼ N(μ,Σ) (regardless of whether Σ is singular or not) iff for any n-dimensional constant vector c, c′X ∼ N(c′μ,c′Σc).

UW-Madison (Statistics)  Stat 609 Lecture 22  2014  10 / 12

beamer-tu-logo

Thus, μ and Σ in N(μ,Σ) is still the mean and covariance matrix. Furthermore, any two components of X ∼ N(μ,Σ) are independent iff they are uncorrelated. This can be shown as follows. Suppose that X_{1} and X_{2} are the first two components of X and Cov(X_{1},X_{2}) = 0, i.e., the (1,2)th and (2,1)th elements of Σ are 0. Let μ_{1} and μ_{2} be the first two components of μ and σ_{1}^{2} and σ_{2}^{2} be the first and second diagonal elements of Σ, and let t = (t_{1},t_{2},0,...,0), t_{1} ∈ R, t_{2} ∈ R. Then the mgf of (X_{1},X_{2}) is

M_{(X_{1},X_{2})}(t_{1},t_{2}) = e^{μ′t+t′Σt/2}= e^{μ_{1}t_{1}+σ_{1}^{2}t_{1}^{2}/2}e^{μ_{2}t_{2}+σ_{2}^{2}t_{2}^{2}/2} t_{1} ∈ R, t_{2} ∈ R

By Theorem M4, X_{1} and X_{2} are independent.

Theorem.

An n-dimensional random vector X ∼ N(μ,Σ) (regardless of whether Σ is singular or not) iff for any n-dimensional constant vector c, c′X ∼ N(c′μ,c′Σc).

UW-Madison (Statistics)  Stat 609 Lecture 22  2014  10 / 12

beamer-tu-logo

Proof.

We treat a degenerated X = c as N(c,0). If X ∼ N(μ,Σ), then M_{X}(t) = e^{μ′t+t′Σt/2}. For any c ∈ R^{n}, by the properties of mgf, the mgf of c′X is M_{c′X}(t) = M_{X}(ct) = e^{μ′(ct)+(ct)′Σ(ct)/2} = e^{(c′μ)t+(c′Σc)t^{2}/2} t ∈ R which is the mgf of N(c′μ,c′Σc). By uniqueness, c′X ∼ N(c′μ,c′Σc). If c′X ∼ N(c′μ,c′Σc) for any c ∈ R^{n}, then t′X ∼ N(t′μ,t′Σt) for any t ∈ R^{n} and

M_{t′X}(s) = e^{(t′μ)s+(t′Σt)s^{2}/2} s ∈ R

Letting s = 1, we obtain

M_{t′X}(1) = e^{(t′μ)+(t′Σt)/2}= E(e^{t′X}) = M_{X}(t) t ∈ R^{n}

By uniqueness, X ∼ N(μ,Σ). The condition any c ∈ R^{n} is important.

UW-Madison (Statistics)  Stat 609 Lecture 22  2014  11 / 12

beamer-tu-logo

The uniform distribution on [a,b] × [c,d]

We have shown that the two marginal distributions are uniform distributions on intervals [a,b] and [c,d]. For non-zero constants ξ and ζ, is the distribution of ξX + ζY a uniform distribution on some interval? If (e^{bt}− e^{at})/t is defined to be b − a when t = 0 for any constants a < b, then

M_{X,Y}(t,s) = ∫_{a}^{b}∫_{c}^{d} e^{tx+sy} \frac{1}{(b − a)(d − c)} dxdy
= \frac{(e^{bt}− e^{at})(e^{ds} − e^{cs})}{(b − a)(d − c)ts}  s,t ∈ R

and

M_{ξX+ζY}(t) = E(e^{t(ξX+ζY )}) = \frac{(e^{bξt}− e^{aξt})(e^{dζt}− e^{cζt})}{(b − a)(d − c)ξζt^{2}} t ∈ R

This is not a mgf of a uniform distribution on an interval [r,h], which is of the form (e^{ht}− e^{rt})/[t(h − r)] for t ∈ R.

UW-Madison (Statistics)  Stat 609 Lecture 22  2014  12 / 12