beamer-tu-logo

Lecture 17: Independence

First definition of independence of random variables/vectors

Let X_{1},...,X_{k} be random variables, F(x_{1},...,x_{k}) be their joint cdf., and F_{X_{i}}(x_{i}) be the marginal cdf of X_{i}, i = 1,...,k. X_{1},...,X_{k} are (statistically) independent iff

F(x_{1},...,x_{k}) = F_{X_{1}}(x_{1})··· F_{X_{k}}(x_{k}) (x_{1},...,x_{k}) ∈ R^{k}

We define the independence of random vectors X_{1},...,X_{k} (they may have different dimensions) in the same way, except that F is the joint cdf of (X_{1},...,X_{k}) and F_{X_{i}} is the (joint) cdf of X_{i}, i = 1,...,k.

Clearly, this definition is equivalent to the following definition.

Second definition of independence of random variables/vectors

Random vectors X_{1},...,X_{k} are independent iff for any permutation i_{1},...,i_{k} of 1,...,k, and any r = 1,...,k − 1, the conditional distribution of X_{i_{1}},...,X_{i_{r}} given X_{i_{r+1}},...,X_{i_{k}} is the same as the distribution of X_{i_{1}},...,X_{i_{r}}.

beamer-tu-logo

Lecture 17: Independence

First definition of independence of random variables/vectors

Let X_{1},...,X_{k} be random variables, F(x_{1},...,x_{k}) be their joint cdf., and F_{X_{i}}(x_{i}) be the marginal cdf of X_{i}, i = 1,...,k. X_{1},...,X_{k} are (statistically) independent iff

F(x_{1},...,x_{k}) = F_{X_{1}}(x_{1})··· F_{X_{k}}(x_{k}) (x_{1},...,x_{k}) ∈ R^{k}

We define the independence of random vectors X_{1},...,X_{k} (they may have different dimensions) in the same way, except that F is the joint cdf of (X_{1},...,X_{k}) and F_{X_{i}} is the (joint) cdf of X_{i}, i = 1,...,k.

Clearly, this definition is equivalent to the following definition.

Second definition of independence of random variables/vectors

Random vectors X_{1},...,X_{k} are independent iff for any permutation i_{1},...,i_{k} of 1,...,k, and any r = 1,...,k − 1, the conditional distribution of X_{i_{1}},...,X_{i_{r}} given X_{i_{r+1}},...,X_{i_{k}} is the same as the distribution of X_{i_{1}},...,X_{i_{r}}.

beamer-tu-logo

The joint cdf is determined by the n marginal cdf’s if X_{1},...,X_{n} are independent; otherwise, the joint cdf depends on marginal cdf’s and conditional distributions. If X_{1},...,X_{k} are independent, then X_{i_{1}},...,X_{i_{r}} are independent for any subest {i_{1},...,i_{r}} ⊂ {1,...,k}. When k = 2 and both X_{1} and X_{2} are discrete random variables, they are independent iff all p_{ij} = p_{i·}p_{·j}in the probability table. If X_{i} has pmf (or pdf) f_{X_{i}}, i = 1,...,k, then X_{1},...,X_{k} are independent iff the joint pmf (or pdf) satisfies

f (x_{1},...,x_{k}) = f_{X_{1}}(x_{1})··· f_{X_{k}}(x_{k}) (x_{1},...,x_{k}) ∈ R^{k}

In fact, sometimes the following lemma can be applied.

Lemma 4.2.7.

Let X and Y be random variables having joint pmf or pdf f (x,y). Then X and Y are independent iff there exist functions g(x) and h(y) such that

f (x,y) = g(x)h(y) x ∈ R, y ∈ R

A similar result holds for any fixed number of random variables/vectors.

UW-Madison (Statistics)  Stat 609 Lecture 17  2014  2 / 11

beamer-tu-logo

The joint cdf is determined by the n marginal cdf’s if X_{1},...,X_{n} are independent; otherwise, the joint cdf depends on marginal cdf’s and conditional distributions. If X_{1},...,X_{k} are independent, then X_{i_{1}},...,X_{i_{r}} are independent for any subest {i_{1},...,i_{r}} ⊂ {1,...,k}. When k = 2 and both X_{1} and X_{2} are discrete random variables, they are independent iff all p_{ij} = p_{i·}p_{·j}in the probability table. If X_{i} has pmf (or pdf) f_{X_{i}}, i = 1,...,k, then X_{1},...,X_{k} are independent iff the joint pmf (or pdf) satisfies

f (x_{1},...,x_{k}) = f_{X_{1}}(x_{1})··· f_{X_{k}}(x_{k}) (x_{1},...,x_{k}) ∈ R^{k}

In fact, sometimes the following lemma can be applied.

Lemma 4.2.7.

Let X and Y be random variables having joint pmf or pdf f (x,y). Then X and Y are independent iff there exist functions g(x) and h(y) such that

f (x,y) = g(x)h(y) x ∈ R, y ∈ R

A similar result holds for any fixed number of random variables/vectors.

UW-Madison (Statistics)  Stat 609 Lecture 17  2014  2 / 11

beamer-tu-logo

In the definition of independence of random vectors, the components of each random vector may be dependent or independent. For example, if X, Y , and Z are 3 random variables, we may have that (X,Y ) and Z are independent but X and Y are not independent.

Example

Suppose that (X,Y ,Z) has pdf

f (x,y,z) = \{ \begin{matrix}e^{−y} e^{−z} & 0 < x < y,z > 0 \\ 0 & otherwise \end{matrix}

It can be shown that f (x,y,z) = g(x,y)h(z) with

g(x,y) = \{ \begin{matrix}e^{−y} & 0 < x < y \\ 0 & otherwise \end{matrix} h(z) = \{ \begin{matrix}e^{−z} & z > 0 \\ 0 & z ≤ 0 \end{matrix}

Hence, Z and U = (X,Y ) are independent. In Example 4.2.4, we showed that Y |X = x ∼ exponential(x,1), The marginal pdf of Y is i.e., gamma(2,1), because

∫_{−∞}^{∞} ∫_{−∞}^{∞} f (x,y,z)dxdz = ∫_{0}^{y} e^{−y} dx = ye^{−y} y > 0
UW-Madison (Statistics)  Stat 609 Lecture 17  2014  3 / 11

beamer-tu-logo

By the 2nd definition of independence, X and Y are not independent. In fact, when Y |X = x has a pdf varying with x, we have already known that X and Y are not independent. Consequently, X, Y , and Z are not independent.

Note that “X, Y , and Z are independent" may be different from “(X,Y ) and Z are independent". Hence, when random vectors are involved, it is important to specify which variables/vectors are independent.

Third definition of independence of random variables/vectors

Random vectors X_{1},...,X_{k} are independent iff events {X_{1} ∈ B_{1}}, ...,{X_{k} ∈ B_{k}} are independent for any Borel sets B_{1},...,B_{k}, where the dimension of B_{i} is the same as the dimension of X_{i}.

It can be shown that this definition is equivalent to the first definition. In fact, it is easy to show this definition implies the first definition, by considering B_{i} = (−∞,x_{i}]. A rigorous proof is out of the scope of this course.

UW-Madison (Statistics)  Stat 609 Lecture 17  2014  4 / 11

beamer-tu-logo

By the 2nd definition of independence, X and Y are not independent. In fact, when Y |X = x has a pdf varying with x, we have already known that X and Y are not independent. Consequently, X, Y , and Z are not independent.

Note that “X, Y , and Z are independent" may be different from “(X,Y ) and Z are independent". Hence, when random vectors are involved, it is important to specify which variables/vectors are independent.

Third definition of independence of random variables/vectors

Random vectors X_{1},...,X_{k} are independent iff events {X_{1} ∈ B_{1}}, ...,{X_{k} ∈ B_{k}} are independent for any Borel sets B_{1},...,B_{k}, where the dimension of B_{i} is the same as the dimension of X_{i}.

It can be shown that this definition is equivalent to the first definition. In fact, it is easy to show this definition implies the first definition, by considering B_{i} = (−∞,x_{i}]. A rigorous proof is out of the scope of this course.

UW-Madison (Statistics)  Stat 609 Lecture 17  2014  4 / 11

beamer-tu-logo

Theorem 4.6.12.

If X_{1},...,X_{k} are independent random vectors and g_{1},...,g_{k} are functions, then

(i) g_{1}(X_{1}),...,g_{k}(X_{k}) are independent, and (ii) E[g_{1}(X_{1})··· g_{k}(X_{k})] = E[g_{1}(X_{1})]··· E[g_{k}(X_{k})].

The following is an important corollary.

Theorem 4.5.5.

If X_{1},...,X_{k} are independent random variables, then X_{i} and X_{j} are uncorrelated for every pair (i,j).

The converse is not true, i.e., there are uncorrelated X and Y but they are not independent. An example is X and Y in Example 4.5.9 with joint pdf

f (x,y) = \{ \begin{matrix}5 & −1 < x < 1, x^{2} < y < x^{2} + 0.1 \\ 0 & otherwise \end{matrix}

We showed that Cov(X,Y ) = 0, i.e., X and Y are uncorrelated.

beamer-tu-logo

Theorem 4.6.12.

If X_{1},...,X_{k} are independent random vectors and g_{1},...,g_{k} are functions, then

(i) g_{1}(X_{1}),...,g_{k}(X_{k}) are independent, and (ii) E[g_{1}(X_{1})··· g_{k}(X_{k})] = E[g_{1}(X_{1})]··· E[g_{k}(X_{k})].

The following is an important corollary.

Theorem 4.5.5.

If X_{1},...,X_{k} are independent random variables, then X_{i} and X_{j} are uncorrelated for every pair (i,j).

The converse is not true, i.e., there are uncorrelated X and Y but they are not independent. An example is X and Y in Example 4.5.9 with joint pdf

f (x,y) = \{ \begin{matrix}5 & −1 < x < 1, x^{2} < y < x^{2} + 0.1 \\ 0 & otherwise \end{matrix}

We showed that Cov(X,Y ) = 0, i.e., X and Y are uncorrelated.

beamer-tu-logo

Theorem 4.6.12.

If X_{1},...,X_{k} are independent random vectors and g_{1},...,g_{k} are functions, then

(i) g_{1}(X_{1}),...,g_{k}(X_{k}) are independent, and (ii) E[g_{1}(X_{1})··· g_{k}(X_{k})] = E[g_{1}(X_{1})]··· E[g_{k}(X_{k})].

The following is an important corollary.

Theorem 4.5.5.

If X_{1},...,X_{k} are independent random variables, then X_{i} and X_{j} are uncorrelated for every pair (i,j).

The converse is not true, i.e., there are uncorrelated X and Y but they are not independent. An example is X and Y in Example 4.5.9 with joint pdf

f (x,y) = \{ \begin{matrix}5 & −1 < x < 1, x^{2} < y < x^{2} + 0.1 \\ 0 & otherwise \end{matrix}

We showed that Cov(X,Y ) = 0, i.e., X and Y are uncorrelated.

beamer-tu-logo

How do we show X and Y are not independent? By the 3rd definition, we just need to find two Borel sets A and B such that

P(X ∈ A,Y ∈ B) ≠ P(X ∈ A)P(Y ∈ B)

It is easy to show that X ∼ uniform(−1,1). The marginal pdf of Y is not simple, but it is unnecessary to derive it for showing that X and Y are not independent. If X and Y are independent, then

P(0 < X < 0.5,0 < Y < 0.1) = P(0 < X < 0.5)P(0 < Y < 0.1)

P(0 < X < \sqrt{0.1},0 < Y < 0.1) = P(0 < X < \sqrt{0.1})P(0 < Y < 0.1) But if 0 < Y < 0.1, we must have |X| < \sqrt{0.1} and, hence,

P(0 < X < 0.5,0 < Y < 0.1) = P(0 < X < \sqrt{0.1},0 < Y < 0.1)

Since the pdf of Y is positive on the interval (0,0.1), P(0 < Y < 1) > 0. This proves that P(0 < X < 0.5) = P(0 < X < \sqrt{0.1}), which is impossible because X ∼ uniform(−1,1). This contradiction shows that X and Y are not independent.

beamer-tu-logo

Interpretation

Uncorrelated X and Y means that there is no linear relationship between X and Y , but X and Y may have a strong nonlinear relationship. Independence of X and Y means that X and Y have no relationship in terms of the occurrence of events related to X and Y (the 2nd definition); hence X and Y don’t have any linear relationship. This example indicates that if the distribution of (X,Y ) is symmetric in some way, then Cov(X,Y ) = 0 and X and Y don’t have any linear relationship.

Example

Let (X,Y ) be a random vector on R^{2} with pdf

f (x,y) = \{ \begin{matrix}π^{−1} & x^{2} + y^{2} ≤ 1 \\ 0 & x^{2} + y^{2} > 1. \end{matrix}

beamer-tu-logo

Interpretation

Uncorrelated X and Y means that there is no linear relationship between X and Y , but X and Y may have a strong nonlinear relationship. Independence of X and Y means that X and Y have no relationship in terms of the occurrence of events related to X and Y (the 2nd definition); hence X and Y don’t have any linear relationship. This example indicates that if the distribution of (X,Y ) is symmetric in some way, then Cov(X,Y ) = 0 and X and Y don’t have any linear relationship.

Example

Let (X,Y ) be a random vector on R^{2} with pdf

f (x,y) = \{ \begin{matrix}π^{−1} & x^{2} + y^{2} ≤ 1 \\ 0 & x^{2} + y^{2} > 1. \end{matrix}

beamer-tu-logo

By changing variables, we obtain that

E(XY ) = π^{−1} ∫_{D}dxdy
= π^{−1} ( ∫_{D,xy>0} dxdy + ∫_{D,xy<0} dxdy)
= π^{−1} ( ∫_{D,xy>0} dxdy − ∫_{D,xy>0} dxdy)
= 0

Similar, EX = EY = 0 and, hence, Cov(X,Y ) = 0. A direct calculation shows that

P(0 < X < 1/\sqrt{2},0 < Y < 1/\sqrt{2}) = \frac{1}{2π}

and

P(0 < X < 1/\sqrt{2}) = P(0 < Y < 1/\sqrt{2}) = \frac{1}{4} + \frac{1}{2π} .

Thus, X and Y are not independent because

\frac{1}{2π} ≠ ( \frac{1}{4} + \frac{1}{2π} ) ^{2}.

UW-Madison (Statistics)  Stat 609 Lecture 17  2014  8 / 11

beamer-tu-logo

Normal random variables

For normally distributed (X,Y ), however, independence of X and Y is equivalent to Cov(X,Y ) = 0. If (X,Y ) is bivariate normal, then the joint pdf is

f (x,y) = \frac{exp( −\frac{(x−μ_{1})^{2}}{2σ_{1}^{2}(1−ρ^{2})} + \frac{ρ(x−μ_{1})(y−μ_{2})}{σ_{1}σ_{2}(1−ρ^{2})} − \frac{(y−μ_{2})^{2}}{2σ_{2}^{2}(1−ρ^{2})} )}{2πσ_{1}σ_{2}\sqrt{1 − ρ^{2}}} (x,y) ∈ R^{2}

We proved that ρ = the correlation between X and Y and Cov(X,Y ) = σ_{1}σ_{2}ρ. If ρ = 0, clearly f (x,y) is a product of a function of x and a function of y; by Lemma 4.2.7, X and Y are independent. On the other hand, if ρ ≠ 0, then X and Y are correlated and they cannot be independent (Theorem 4.5.5).

This indicates that for normally distributed (X,Y ), either X and Y have a linear relationship or they don’t have any relationship at all (they cannot have a nonlinear relationship).

beamer-tu-logo

In some applications, the concept of independence of X and Y is not enough. There are situations in which X and Y are not independent, but if some information regarding another random vector Z is given, then X and Y are conditionally independent.

Definition (conditional Independence)

Let X, Y , and Z be random vectors. X and Y are conditionally independent given Z iff

F(x,y|z) = F(x|z)F(y|z) for all x, y, z

where F(x,y|z) is the cdf of (X,Y )|Z = z, F(x|z) is the cdf of X|Z = z, and F(y|z) is the cdf of Y |Z = z.

Example

Let Z, U, and V be independent random variables, X = Z + U, and Y = Z + V . Then X and Y are not independent because of the common term Z.

beamer-tu-logo

In some applications, the concept of independence of X and Y is not enough. There are situations in which X and Y are not independent, but if some information regarding another random vector Z is given, then X and Y are conditionally independent.

Definition (conditional Independence)

Let X, Y , and Z be random vectors. X and Y are conditionally independent given Z iff

F(x,y|z) = F(x|z)F(y|z) for all x, y, z

where F(x,y|z) is the cdf of (X,Y )|Z = z, F(x|z) is the cdf of X|Z = z, and F(y|z) is the cdf of Y |Z = z.

Example

Let Z, U, and V be independent random variables, X = Z + U, and Y = Z + V . Then X and Y are not independent because of the common term Z.

beamer-tu-logo

For example, if E(XY ) exists, then

Cov(X,Y ) = Cov(Z + U,Z + V )
= Cov(Z,Z) + Cov(Z,U) + Cov(Z,V ) + Cov(U,V )
= Var(Z) > 0

But conditioned on Z = z, X and Y are independent because X = z + U and Y = z + V and U and V are independent. A rough proof is the following.

F(x − z,y − z|z) = F(u,v|z) = F(u,v) = F(u)F(v)
= F(u|z)F(v|z) = F(x − z|z)F(y − z|z)

This holds for all x, y, and z, and hence

F(x,y|z) = F(x|z)F(y|z)