beamer-tu-logo

Lecture 23: Multivariate normal distributions (part II)

We have shown that if X ∼ N(μ,Σ), then any linear function AX + b is normally distributed. The following result concerns the independence of linear functions of a normally distributed random vector.

Theorem N1.

Let X be an n-dimensional random vector ∼ N(μ,Σ) and A be a fixed k × n matrix, and B be a fixed l × n matrix. Then, AX and BX are independent iff AΣB′= 0.

Proof.

Let

Y = ( \begin{matrix}A \\ B \end{matrix} ) X = ( \begin{matrix}AX \\ BX \end{matrix} )

From the properties of the multivariate normal distribution, we know that Y is multivariate normal with covariance matrix

( \begin{matrix}A \\ B \end{matrix} ) Σ(A′ B′) = ( \begin{matrix}AΣA′ & AΣB′ \\ BΣA′ & BΣB′ \end{matrix} )
UW-Madison (Statistics)  Stat 609 Lecture 23  2014  1 / 12

beamer-tu-logo

Hence, AX and BX are uncorrelated iff AΣB′= 0 and, thus, the only if part follows since independence implies no correlation. The proof for the if part is the same as the proof of two uncorrelated components of X are independent: we can show that if AΣB′= 0, then the mgf of (AX,BX) is a product of an mgf on R^{k} and another mgf on R^{l}, and then apply Theorem M4.

Theorem N2.

If (X,Y ) is a random vector ∼ N(μ,Σ) with

μ = ( \begin{matrix}μ_{1} \\ μ_{2} \end{matrix} ) , Σ = ( \begin{matrix}Σ_{11} & Σ_{12} \\ Σ_{21} & Σ_{22} \end{matrix} ) ,

and if Σ is positive definite, then

Y |X ∼ N ( μ_{2} + Σ_{21}Σ_{11}^{−1} (X − μ_{1}),Σ_{22} − Σ_{21}Σ_{11}^{−1} Σ_{12})

It follows from the properties of normal distributions that

E(Y |X) = μ_{2} + Σ_{21}Σ_{11}^{−1} (X − μ_{1}), Var(Y |X) = Σ_{22} − Σ_{21}Σ_{11}^{−1} Σ_{12} While the conditional mean depends on X, the conditional covariance matrix does not.

UW-Madison (Statistics)  Stat 609 Lecture 23  2014  2 / 12

beamer-tu-logo

Hence, AX and BX are uncorrelated iff AΣB′= 0 and, thus, the only if part follows since independence implies no correlation. The proof for the if part is the same as the proof of two uncorrelated components of X are independent: we can show that if AΣB′= 0, then the mgf of (AX,BX) is a product of an mgf on R^{k} and another mgf on R^{l}, and then apply Theorem M4.

Theorem N2.

If (X,Y ) is a random vector ∼ N(μ,Σ) with

μ = ( \begin{matrix}μ_{1} \\ μ_{2} \end{matrix} ) , Σ = ( \begin{matrix}Σ_{11} & Σ_{12} \\ Σ_{21} & Σ_{22} \end{matrix} ) ,

and if Σ is positive definite, then

Y |X ∼ N ( μ_{2} + Σ_{21}Σ_{11}^{−1} (X − μ_{1}),Σ_{22} − Σ_{21}Σ_{11}^{−1} Σ_{12})

It follows from the properties of normal distributions that

E(Y |X) = μ_{2} + Σ_{21}Σ_{11}^{−1} (X − μ_{1}), Var(Y |X) = Σ_{22} − Σ_{21}Σ_{11}^{−1} Σ_{12} While the conditional mean depends on X, the conditional covariance matrix does not.

UW-Madison (Statistics)  Stat 609 Lecture 23  2014  2 / 12

beamer-tu-logo

Proof.

Consider the transformation

U = AX + Y

with a fixed matrix A chosen so that U and X are independent. From Theorem N1, we need U and X to be uncorrelated. Since

Cov(X,U) = Cov(X,AX + Y ) = Cov(X,AX) + Cov(X,Y )
= Cov(X,X)A′+ Σ_{12} = Σ_{11}A′+ Σ_{12}

we choose A = −Σ_{21}Σ_{11}^{−1} . Consider the transformation

( \begin{matrix}V \\ U \end{matrix} ) = ( \begin{matrix}X \\ AX + Y \end{matrix} ) = ( \begin{matrix}I & 0 \\ −Σ_{21}Σ_{11}^{−1} & I \end{matrix} ) ( \begin{matrix}X \\ Y \end{matrix} ) ,  | \frac{∂(U,V )}{∂(X,Y )}| = 1

Let f_{(X,Y}) be the pdf of (X,Y ), f_{(U,V )} be the pdf of (U,V ), f_{U} be the pdf of U and f_{V} be the pdf of V . By the transformation formula and the independence of U and V = X,

UW-Madison (Statistics)  Stat 609 Lecture 23  2014  3 / 12

beamer-tu-logo

f_{(X,Y )}(x,y) = f_{(U,V )}(u,v) = f_{U}(u)f_{V}(v) = f_{U}(y − Σ_{21}Σ_{11}^{−1} x)f_{X}(x)

Then the pdf of Y |X is

\frac{f_{(X,Y )}(x,y)}{f_{X}(x)} = \frac{f_{U}(y − Σ_{21}Σ_{11}^{−1} x)f_{X}(x)}{f_{X}(x)} = f_{U}(y − Σ_{21}Σ_{11}^{−1} x)

Since U = −Σ_{21}Σ_{11}^{−1} X + Y , U is normally distributed.

E(U) = −Σ_{21}Σ_{11}^{−1} E(X) + E(Y ) = −Σ_{21}Σ_{11}^{−1} μ_{1} + μ_{2}

Var(U) = Var(AX + Y ) = Var(AX) + Var(Y ) + 2Cov(AX,Y )
= AVar(X)A′+ Σ_{22} + 2ACov(X,Y )
= Σ_{21}Σ_{11}^{−1} Σ_{11}Σ_{11}^{−1} Σ_{12} + Σ_{22} − 2Σ_{21}Σ_{11}^{−1} Σ_{12}
= Σ_{22} − Σ_{21}Σ_{11}^{−1} Σ_{12}

Hence, f_{U} is the pdf of N(μ_{2} − Σ_{21}Σ_{11}^{−1} μ_{1},Σ_{22} − Σ_{21}Σ_{11}^{−1} Σ_{12}). Given X = x, Σ_{21}Σ_{11}^{−1} x is a constant and, hence, f_{U}(y − Σ_{21}Σ_{11}^{−1} x) is the pdf of N(μ_{2} + Σ_{21}Σ_{11}^{−1} (x − μ_{1}),Σ_{22} − Σ_{21}Σ_{11}^{−1} Σ_{12}), considered as a function of y.

UW-Madison (Statistics)  Stat 609 Lecture 23  2014  4 / 12

beamer-tu-logo

Quadratic forms

For a random vector X and a fixed symmetric matrix A, X′AX is called a quadratic function or quadratic form of X. We now study the distribution of quadratic forms when X is multivariate normal.

Theorem N3.

Let X ∼ N(μ,I_{n}) and A be a fixed n × n symmetric matrix. A necessary and sufficient condition for X′AX is chi-square distributed is A^{2} = A, in which case the degrees of freedom of the chi-square distribution is the rank of A and the noncentrality parameter μ′Aμ.

Proof.

Sufficiency. If A^{2} = A, then A is a projection matrix and there exists an n × n matrix T such that T′T = TT′= I_{n} and

A = T′( \begin{matrix}I_{k} & 0 \\ 0 & 0 \end{matrix} ) T = C′C

beamer-tu-logo

Quadratic forms

For a random vector X and a fixed symmetric matrix A, X′AX is called a quadratic function or quadratic form of X. We now study the distribution of quadratic forms when X is multivariate normal.

Theorem N3.

Let X ∼ N(μ,I_{n}) and A be a fixed n × n symmetric matrix. A necessary and sufficient condition for X′AX is chi-square distributed is A^{2} = A, in which case the degrees of freedom of the chi-square distribution is the rank of A and the noncentrality parameter μ′Aμ.

Proof.

Sufficiency. If A^{2} = A, then A is a projection matrix and there exists an n × n matrix T such that T′T = TT′= I_{n} and

A = T′( \begin{matrix}I_{k} & 0 \\ 0 & 0 \end{matrix} ) T = C′C

beamer-tu-logo

Then X′AX = (CT )′(CT ) is simply the sum of the squares of the first k components of TX. Since TX ∼ N(T μ,TI_{n}T′) = N(T μ,I_{n}), by definition X′AX has the chi-square distribution with degrees of freedom k and noncentrality parameter (Cμ)′(Cμ) = μ′C′Cμ = μ′Aμ.

Necessity. Suppose that X′AX is chi-square with degrees of freedom m and noncentrality parameter δ ≥ 0. Then A must be nonnegative definite and there exists an n × n matrix T such that T′T = TT′= I_{n} and

A = T′( \begin{matrix}Λ & 0 \\ 0 & 0 \end{matrix} ) T

where Λ is a k × k diagonal matrix contains k non-zero eigenvalues 0 < λ_{1} ≤ ···≤ λ_{k}. We still have TX ∼ N(T μ,I_{n}). Let Y_{1},...,Y_{k} be the first k components of TX. Then Y_{i}^{2}’s are independent and Y_{i}^{2} ∼ chi-square with degree of freedom 1 and noncentrality parameter μ_{i}^{2}, where μ_{i} is the ith

UW-Madison (Statistics)  Stat 609 Lecture 23  2014  6 / 12

beamer-tu-logo

component of μ, and

X′AX = ∑_{i=1}^{k}λ_{i}Y_{i}^{2}

Using the mgf formula for noncentral chi-square distributions, the mgf’s of the left and right hand sides are respectively given in the left and right hand sides of the following:

\frac{e^{δt/(1−2t)}}{(1 − 2t)^{m/2}} = ∏_{i=1}^{k} \frac{e^{λ_{i}μ_{i}^{2}t/(1−2λ_{i}t)}}{(1 − 2λ_{i}t)^{1/2}} t < 1/2

Suppose that λ_{k} > 1. When t → (2λ_{k})^{−1} , the right hand side of the above equation diverges to ∞ whereas the left hand side of the above equation goes to e^{δ(2λ_{k})^{−1} /(1−λ_{k}^{−1} )} /(1 − λ_{k}^{−1} )^{m/2} < ∞, which is a contradiction. Hence λ_{k} ≤ 1 so that λ_{i} ≤ 1 for all i. Suppose that λ_{k} = ···= λ_{l+1} = 1 > λ_{l} ≥ ···≥ λ_{1} > 0 for a positive integer l ≤ k, which implies

\frac{e^{δt/(1−2t)}}{(1 − 2t)^{(m−k+l)/2}} = ∏_{i=1}^{l} \frac{e^{λ_{i}μ_{i}^{2}t/(1−2λ_{i}t)}}{(1 − 2λ_{i}t)^{1/2}} t < 1/2

UW-Madison (Statistics)  Stat 609 Lecture 23  2014  7 / 12

beamer-tu-logo

When t → 1/2, the left hand side of the above equation diverges to ∞, whereas the right hand side of the above equation converges to

∏_{i=1}^{l} \frac{e^{λ_{i}μ_{i}^{2}/2(1−λ_{i})}}{(1 − λ_{i})^{1/2}}

which is a contradiction. Therefore, we must have λ_{1} = ···= λ_{k} = 1, i.e., A is a projection matrix.

Theorem N4 (Cochran’s theorem).

Suppose that X is an n-dimensional random vector ∼ N(μ,I_{n}) and

X′X = X′A_{1}X + ··· + X′A_{k}X,

where I_{n} is the n × n identity matrix and A_{i} is an n × n symmetric matrix with rank n_{i}, i = 1,...,k. A necessary and sufficient condition for

(i) X′A_{i}X has the noncentral chi-square distribution with degrees of freedom n_{i} and noncentrality parameter δ_{i}, i = 1,...,k,

(ii) X′A_{i}X’s are independent, is n = n_{1} + ··· + n_{k}, in which case δ_{i} = μ′A_{i}μ and δ_{1} + ··· + δ_{k} = μ′μ.

UW-Madison (Statistics)  Stat 609 Lecture 23  2014  8 / 12

beamer-tu-logo

Proof.

Suppose that (i)-(ii) hold. Then X′X has the chi-square distribution with degrees of freedom n_{1} + ··· + n_{k} and noncentrality parameter δ_{1} + ··· + δ_{k}. By definition, X′X has the noncentral chi-square distribution with degrees of freedom n and noncentrality parameter μ′μ. Then we must have n = n_{1} + ··· + n_{k} and δ_{1} + ··· + δ_{k} = μ′μ. Suppose now that n = n_{1} + ··· + n_{k}. From the theory of linear algebra, for each i there exists c_{ij} ∈ R^{n}, j = 1,...,n_{i}, such that

X′A_{i}X = ±(c′_{i1}X)^{2} ± ··· ± (c′_{in_{i}}X)^{2}

Let C be the n ×n matrix whose columns are c_{11},...,c_{1n_{1}},...,c_{k1},...,c_{kn_{k}}, Then

X′X = X′CΔC′X

with an n × n diagonal matrix Δ whose diagonal elements are ±1. This implies CΔC′= I_{n} and thus C is of full rank and Δ = C^{−1} (C′)^{−1} , which is positive definite.

UW-Madison (Statistics)  Stat 609 Lecture 23  2014  9 / 12

beamer-tu-logo

This shows Δ = I_{n}, which implies C′C = CC′= I_{n} and

X′A_{i}X = ∑_{j=n_{1}+··· +n_{i−1} +1}^{n_{1}+··· +n_{i−1} +n_{i}}Y_{j}^{2},

where Y_{j} is the jth component of Y = C′X ∼ N(C′μ,I_{n}). Hence Y_{j}’s are independent and Y_{j} ∼ N(λ_{j},1), where λ_{j} is the jth component of C′μ. This shows that X′A_{i}X, i = 1,...,k, are independent and X′A_{i}X has the chi-square distribution with degrees of freedom n_{i} and noncentrality parameter δ_{i} = λ_{n_{1}+··· +n_{i−1} +1}^{2} + ··· + λ_{n_{1}+··· +n_{i−1} +n_{i}}^{2}.

Letting X = μ and Y = C′X = C′μ, we obtain that δ_{i} = μ′A_{i}μ and δ_{1} + ··· + δ_{k} = μ′CC′μ = μ′μ. This completes the proof.

Theorem N5.

Let X be an n-dimensional random vector ∼ N(μ,I_{n}) and A_{1} and A_{2} be n × n projection matrices. Then a necessary and sufficient condition that X′A_{1}X and X′A_{2}X are independent is A_{1}A_{2} = 0.

UW-Madison (Statistics)  Stat 609 Lecture 23  2014  10 / 12

beamer-tu-logo

Proof.

If A_{1}A_{2} = 0, then

(I_{n}− A_{1} − A_{2})^{2} = I_{n}− A_{1} − A_{2} − A_{1} + A_{1}^{2} + A_{2}A_{1} − A_{2} + A_{1}A_{2} + A_{2}^{2}
= I_{n}− A_{1} − A_{2},

i.e., I_{n}− A_{1} − A_{2} is a projection matrix with rank = trace(I_{n}− A_{1} − A_{2}) = n − r_{1} − r_{2}, where r_{i} = trace(A_{i}) is the rank of A_{i}, i = 1,2. By Cochran’s theorem and

X′X = X′A_{1}X + X′A_{2}X + X′(I_{n}− A_{1} − A_{2})X,

X′A_{1}X and X′A_{2}X are independent. This proves the sufficiency. Assume that X′A_{1}X and X′A_{2}X are independent. Since X′A_{i}X has the noncentral chi-square distribution with degrees of freedom r_{i} = the rank of A_{i} and noncentrality parameter δ_{i} = μ′A_{i}μ, X′(A_{1} + A_{2})X has the noncentral chi-square distribution with degrees of freedom r_{1} + r_{2} and noncentrality parameter δ_{1} + δ_{2}.

beamer-tu-logo

Consequently, A_{1} + A_{2} is a projection matrix, i.e.,

(A_{1} + A_{2})^{2} = A_{1} + A_{2},

which implies

A_{1}A_{2} + A_{2}A_{1} = 0.

Since A_{1}^{2} = A_{1}, we obtain that

0 = A_{1}(A_{1}A_{2} + A_{2}A_{1}) = A_{1}A_{2} + A_{1}A_{2}A_{1}

and

0 = A_{1}(A_{1}A_{2} + A_{2}A_{1})A_{1} = 2A_{1}A_{2}A_{1},

which imply A_{1}A_{2} = 0. This proves the necessity.