beamer-tu-logo

Lecture 16: Conditional distribution and expectation

Definition 4.2.1 (conditional pmf)

Let (X_{1},...,X_{n}) be a discrete random vector with joint pmf f (x) and k be an integer satisfying 1 ≤ k ≤ n − 1. The conditional pmf of (X_{k+1},...,X_{n}) given that (X_{1},...,X_{k}) = (x_{1},...,x_{k}) is

f (x_{k+1},...,x_{n}|x_{1},...,x_{k}) = P(X_{k+1} = x_{k+1},...,X_{n} = x_{n}|X_{1} = x_{1},...,X_{k} = x_{k})

= (f (x_{1},...,x_{n})) / (∑_{(y_{k+1},...,y_{n})∈N_{k}} f (x_{1},...,x_{k},y_{k+1},...,y_{n}))

where N_{k} = {(y_{k+1},...,y_{n}) : P(X_{k+1} = y_{k+1},...X_{n} = y_{n}) > 0} and it is assumed that P(X_{1} = x_{1},...,X_{k} = x_{k}) > 0.

It can be easily verified that f (x_{k+1},...,x_{n}|x_{1},...,x_{k}) is a pmf for any

(x_{1},...,x_{k}) with P(X_{1} = x_{1},...,X_{k} = x_{k}) > 0.
The conditional pmf f (x_{k+1},...,x_{n}|x_{1},...,x_{k}) vary with x_{1},...,x_{k}.

beamer-tu-logo

Lecture 16: Conditional distribution and expectation

Definition 4.2.1 (conditional pmf)

Let (X_{1},...,X_{n}) be a discrete random vector with joint pmf f (x) and k be an integer satisfying 1 ≤ k ≤ n − 1. The conditional pmf of (X_{k+1},...,X_{n}) given that (X_{1},...,X_{k}) = (x_{1},...,x_{k}) is

f (x_{k+1},...,x_{n}|x_{1},...,x_{k}) = P(X_{k+1} = x_{k+1},...,X_{n} = x_{n}|X_{1} = x_{1},...,X_{k} = x_{k})

= (f (x_{1},...,x_{n})) / (∑_{(y_{k+1},...,y_{n})∈N_{k}} f (x_{1},...,x_{k},y_{k+1},...,y_{n}))

where N_{k} = {(y_{k+1},...,y_{n}) : P(X_{k+1} = y_{k+1},...X_{n} = y_{n}) > 0} and it is assumed that P(X_{1} = x_{1},...,X_{k} = x_{k}) > 0.

It can be easily verified that f (x_{k+1},...,x_{n}|x_{1},...,x_{k}) is a pmf for any

(x_{1},...,x_{k}) with P(X_{1} = x_{1},...,X_{k} = x_{k}) > 0.
The conditional pmf f (x_{k+1},...,x_{n}|x_{1},...,x_{k}) vary with x_{1},...,x_{k}.

beamer-tu-logo

For any event A ⊂ R^{n−k} ,

P((X_{k+1},...,X_{n}) ∈ A|X_{1} = x_{1},...,X_{k} = x_{k})

= ∑_{(x_{k+1},...,x_{n})∈A}f (x_{k+1},...,x_{n}|x_{1},...,x_{k})

When n = 2 and k = 1, X_{1} = X, X_{2} = Y , f_{X}(x) is the marginal pmf of X, the conditional pmf of Y given X = x is

f (y|x) = \frac{f (x,y)}{f_{X}(x)}

Example 4.2.2.

Consider a discrete (X,Y ) having joint pmf given by the following table:

y

10 20 30 f_{X}
0 1/9 1/9 0 2/9
x 1 1/6 2/9 1/6 5/9
2 0 0 2/9 2/9
f_{Y} 5/18 1/3 7/18 1

UW-Madison (Statistics)  Stat 609 Lecture 16  2014  2 / 11

When x = 0,

f (10|0) = f (20|0) = \frac{1/9}{2/9} = \frac{1}{2} f (30|0) = \frac{0}{2/9} = 0

When x = 1,

f (10|1) = f (30|1) = \frac{1/6}{5/9} = \frac{3}{10} f (20|1) = \frac{2/9}{5/9} = \frac{2}{5}

When x = 2,

f (10|2) = f (20|2) = \frac{0}{2/9} = 0 f (30|2) = \frac{2/9}{2/9} = 1

Definition 4.2.3 (conditional pdf)

Let (X_{1},...,X_{n}) be a random vector with joint pdf f (x) and k be an integer satisfying 1 ≤ k ≤ n − 1. The conditional pdf of (X_{k+1},...,X_{n}) given that (X_{1},...,X_{k}) = (x_{1},...,x_{k}) is

f (x_{k+1},...,x_{n}|x_{1},...,x_{k}) = \frac{f (x_{1},...,x_{n})}{_{∫R^{n−k}} f (x_{1},...,x_{k},y_{k+1},...,y_{n})dy_{k+1}··· dy_{n}}

assuming that the denominator is not 0.

beamer-tu-logo UW-Madison (Statistics)  Stat 609 Lecture 16  2014  3 / 11

f (10|0) = f (20|0) = \frac{1/9}{2/9} = \frac{1}{2} f (30|0) = \frac{0}{2/9} = 0

When x = 1,

f (10|1) = f (30|1) = \frac{1/6}{5/9} = \frac{3}{10} f (20|1) = \frac{2/9}{5/9} = \frac{2}{5}

When x = 2,

f (10|2) = f (20|2) = \frac{0}{2/9} = 0 f (30|2) = \frac{2/9}{2/9} = 1

Definition 4.2.3 (conditional pdf)

Let (X_{1},...,X_{n}) be a random vector with joint pdf f (x) and k be an integer satisfying 1 ≤ k ≤ n − 1. The conditional pdf of (X_{k+1},...,X_{n}) given that (X_{1},...,X_{k}) = (x_{1},...,x_{k}) is

f (x_{k+1},...,x_{n}|x_{1},...,x_{k}) = \frac{f (x_{1},...,x_{n})}{_{∫R^{n−k}} f (x_{1},...,x_{k},y_{k+1},...,y_{n})dy_{k+1}··· dy_{n}}

assuming that the denominator is not 0.

beamer-tu-logo UW-Madison (Statistics)  Stat 609 Lecture 16  2014  3 / 11

beamer-tu-logo

It can be easily verified that f (x_{k+1},...,x_{n}|x_{1},...,x_{k}) is a pdf and for any event A ⊂ R^{n−k} ,

P((X_{k+1},...,X_{n}) ∈ A|X_{1} = x_{1},...,X_{k} = x_{k})
= ∫_{A}f (x_{k+1},...,x_{n}|x_{1},...,x_{k})dx_{k+1}··· dx_{n}

In general, for random vectors X and Y (discrete or conditions), we use the notation Y |X = x or Y |X to denote the conditional distribution Y given X = x or given X.

Example 4.2.4.

Suppose that a 2-dimensional random vector (X,Y ) has pdf

f (x,y) = \{ \begin{matrix}e^{−y} & 0 < x < y \\ 0 & otherwise \end{matrix}

The marginal pdf of X is

f_{X}(x) = ∫_{−∞}^{∞} f (x,y)dy = ∫_{x}^{∞} e^{−y} dy = e^{−x}

beamer-tu-logo

It can be easily verified that f (x_{k+1},...,x_{n}|x_{1},...,x_{k}) is a pdf and for any event A ⊂ R^{n−k} ,

P((X_{k+1},...,X_{n}) ∈ A|X_{1} = x_{1},...,X_{k} = x_{k})
= ∫_{A}f (x_{k+1},...,x_{n}|x_{1},...,x_{k})dx_{k+1}··· dx_{n}

In general, for random vectors X and Y (discrete or conditions), we use the notation Y |X = x or Y |X to denote the conditional distribution Y given X = x or given X.

Example 4.2.4.

Suppose that a 2-dimensional random vector (X,Y ) has pdf

f (x,y) = \{ \begin{matrix}e^{−y} & 0 < x < y \\ 0 & otherwise \end{matrix}

The marginal pdf of X is

f_{X}(x) = ∫_{−∞}^{∞} f (x,y)dy = ∫_{x}^{∞} e^{−y} dy = e^{−x}

beamer-tu-logo

For each x > 0,

f (y|x) = \frac{f (x,y)}{f_{X}(x)} = \{ \begin{matrix}\frac{e^{−y}}{e^{−x}} = e^{−(y−x)} & y > x \\ 0 & y ≤ x \end{matrix}

i.e., f (y|x) is the pdf of exponential(x,1) for x > 0. Thus, Y |X = x ∼ exponential(x,1) or Y |X ∼ exponential(X,1).

Conditional expectations

Let (X_{1},...,X_{n}) be a random vector with joint pmf or pdf f (x), k be an integer satisfying 1 ≤ k ≤ n − 1, and g be a function on R^{n−k} . The conditional expectation of g(X_{k+1},...,X_{n}) given (X_{1},...,X_{k}) = (x_{1},...,x_{k}) is

E[g(X_{k+1},...,X_{n})|X_{1} = x_{1},...,X_{k} = x_{k}]

= ∑_{x_{k+1},...,x_{n}}g(x_{k+1},...,x_{n})f (x_{k+1},...,x_{n}|x_{1},...,x_{k})

when f is a pmf and

= ∫_{R^{n−k}} g(x_{k+1},...,x_{n})f (x_{k+1},...,x_{n}|x_{1},...,x_{k})dx_{k+1}··· dx_{n}

beamer-tu-logo

f (y|x) = \frac{f (x,y)}{f_{X}(x)} = \{ \begin{matrix}\frac{e^{−y}}{e^{−x}} = e^{−(y−x)} & y > x \\ 0 & y ≤ x \end{matrix}

i.e., f (y|x) is the pdf of exponential(x,1) for x > 0. Thus, Y |X = x ∼ exponential(x,1) or Y |X ∼ exponential(X,1).

Conditional expectations

Let (X_{1},...,X_{n}) be a random vector with joint pmf or pdf f (x), k be an integer satisfying 1 ≤ k ≤ n − 1, and g be a function on R^{n−k} . The conditional expectation of g(X_{k+1},...,X_{n}) given (X_{1},...,X_{k}) = (x_{1},...,x_{k}) is

E[g(X_{k+1},...,X_{n})|X_{1} = x_{1},...,X_{k} = x_{k}]

= ∑_{x_{k+1},...,x_{n}}g(x_{k+1},...,x_{n})f (x_{k+1},...,x_{n}|x_{1},...,x_{k})

when f is a pmf and

= ∫_{R^{n−k}} g(x_{k+1},...,x_{n})f (x_{k+1},...,x_{n}|x_{1},...,x_{k})dx_{k+1}··· dx_{n}

beamer-tu-logo

The condition expectation E[g(X_{k+1},...,X_{n})|X_{1} = x_{1},...,X_{k} = x_{k}] is a function of x_{1},...,x_{k}. It is an expectation of the conditional distribution. Let h(x_{1},...,x_{k}) = E[g(X_{k+1},...,X_{n})|X_{1} = x_{1},...,X_{k} = x_{k}]. Then h(X_{1},...,X_{k}) is a random variable and is denoted by E[g(X_{k+1},...,X_{n})|X_{1},...,X_{k}].

Example 4.2.4.

Since Y |X ∼ exponential(X,1), the conditional expectation of Y given X is E(Y |X) = 1 + X, and the conditional expectation of [Y − E(Y |X)]^{2} given X is

E{[Y − E(Y |X)]^{2}|X} = ∫_{X}^{∞} [y − (1 + X)]^{2}e^{−y} dy
= ∫_{X}^{∞} y^{2}e^{−y} dy − (1 + X)^{2}
= 1 + (1 + X)^{2} − (1 + X)^{2} = 1

beamer-tu-logo

The condition expectation E[g(X_{k+1},...,X_{n})|X_{1} = x_{1},...,X_{k} = x_{k}] is a function of x_{1},...,x_{k}. It is an expectation of the conditional distribution. Let h(x_{1},...,x_{k}) = E[g(X_{k+1},...,X_{n})|X_{1} = x_{1},...,X_{k} = x_{k}]. Then h(X_{1},...,X_{k}) is a random variable and is denoted by E[g(X_{k+1},...,X_{n})|X_{1},...,X_{k}].

Example 4.2.4.

Since Y |X ∼ exponential(X,1), the conditional expectation of Y given X is E(Y |X) = 1 + X, and the conditional expectation of [Y − E(Y |X)]^{2} given X is

E{[Y − E(Y |X)]^{2}|X} = ∫_{X}^{∞} [y − (1 + X)]^{2}e^{−y} dy
= ∫_{X}^{∞} y^{2}e^{−y} dy − (1 + X)^{2}
= 1 + (1 + X)^{2} − (1 + X)^{2} = 1

The bivariate normal distribution

Consider the bivariate normal distribution pdf on R^{2}:

f (x,y) = \frac{exp( −\frac{(x−μ_{1})^{2}}{2σ_{1}^{2}(1−ρ^{2})} + \frac{ρ(x−μ_{1})(y−μ_{2})}{σ_{1}σ_{2}(1−ρ^{2})} − \frac{(y−μ_{2})^{2}}{2σ_{2}^{2}(1−ρ^{2})} )}{2πσ_{1}σ_{2}\sqrt{1 − ρ^{2}}} (x,y) ∈ R^{2}

We have shown that the marginal pdf for X is N(μ_{1},σ_{1}^{2}). Hence

\frac{f (x,y)}{f_{X}(x)} = \frac{exp( −\frac{(x−μ_{1})^{2}}{2σ_{1}^{2}(1−ρ^{2})} + \frac{ρ(x−μ_{1})(y−μ_{2})}{σ_{1}σ_{2}(1−ρ^{2})} − \frac{(y−μ_{2})^{2}}{2σ_{2}^{2}(1−ρ^{2})} )}{2πσ_{1}σ_{2}\sqrt{1 − ρ^{2}}} / \frac{exp( −\frac{(x−μ_{1})^{2}}{2σ_{1}^{2}} )}{\sqrt{2π}σ_{1}}

which is equal to

\frac{exp( −\frac{ρ^{2}(x−μ_{1})^{2}}{2σ_{1}^{2}(1−ρ^{2})} + \frac{ρ(x−μ_{1})(y−μ_{2})}{σ_{1}σ_{2}(1−ρ^{2})} − \frac{(y−μ_{2})^{2}}{2σ_{2}^{2}(1−ρ^{2})} )}{\sqrt{2π}σ_{2}\sqrt{1 − ρ^{2}}} = \frac{exp( −\frac{[y−μ_{2}− \frac{ρσ_{2}}{σ_{1}} (x−μ_{1})]^{2}}{2σ_{2}^{2}(1−ρ^{2})} )}{\sqrt{2π}σ_{2}\sqrt{1 − ρ^{2}}}

Hence,

Y |X ∼ N ( μ_{2} + \frac{ρσ_{2}}{σ_{1}} (X − μ_{1}),σ_{2}^{2}(1 − ρ^{2}))

beamer-tu-logo UW-Madison (Statistics)  Stat 609 Lecture 16  2014  7 / 11

beamer-tu-logo

Properties of conditional expectations

Conditional expectations have the following useful properties. Let X, Y , and Z be random variables.

1 If P(Y = c) = 1 for a constant c, then E(Y |X) = c. 2 If Y ≤ Z, then E(Y |X) ≤ E(Z|X). 3 For constants a and b, E(aY + bZ|X) = aE(Y |X) + bE(Z|X). 4 E[E(Y |X)] = E(Y ) (Theorem 4.4.3). This can be interpreted as:

the average of averages is the overall average. 5 Var(Y ) = E[Var(Y |X)] + Var(E(Y |X)) (Theorem 4.4.7), where

Var(Y |X) is the variance of the conditional distribution Y |X. 6 For any function g(X), E[Yg(X)|X] = g(X)E(Y |X).

Proof of Theorem 4.4.3.

Consider the continuous case where (X,Y ) has pdf f (x,y).

E(Y ) = ∫_{−∞}^{∞} yf_{Y}(y)dy = ∫_{−∞}^{∞} y [∫_{−∞}^{∞} f (x,y)]dy
= ∫_{−∞}^{∞} y [∫_{−∞}^{∞} f (x,y)dx]dy = ∫_{−∞}^{∞} ∫_{−∞}^{∞} yf (x,y)dxdy
UW-Madison (Statistics)  Stat 609 Lecture 16  2014  8 / 11

beamer-tu-logo

= ∫_{−∞}^{∞} [∫_{−∞}^{∞} yf (x,y)dy]dx = ∫_{−∞}^{∞} [∫_{−∞}^{∞} y \frac{f (x,y)}{f_{X}(x)}dy]f_{X}(x)dx
= ∫_{−∞}^{∞} [∫_{−∞}^{∞} yf (y|x)dy]f_{X}(x)dx = ∫_{−∞}^{∞} E(Y |X = x)f_{X}(x)dx
= E[E(Y |X)]

Proof of Theorem 4.4.7.

Using properties 1, 3, 4, and 6, we obtain

Var(Y ) = E[Y − E(Y )]^{2} = E ( E{[Y − E(Y )]^{2}|X})
= E ( E{[Y − E(Y |X) + E(Y |X) − E(Y )]^{2}|X})
= E ( E{[Y − E(Y |X)]^{2}|X}) + E ( E{[E(Y |X) − E(Y )]^{2}|X})

+2E (E{[Y − E(Y |X)][E(Y |X) − E(Y )]|X})

= E (Var(Y |X))+ E ( [E(Y |X) − E(Y )]^{2})

+2E ([E(Y |X) − E(Y )]E{[Y − E(Y |X)]|X})

= E[Var(Y |X)] + Var(E(Y |X))
UW-Madison (Statistics)  Stat 609 Lecture 16  2014  9 / 11

beamer-tu-logo

Except for property 2, all properties can be extended to random vectors X, Y , and Z with appropriate modifications on vector multiplications.

Conditional mean and covariance matrix for random vectors

In general, if X and Y are random vectors, then the conditional mean and covariance matrix of Y given X are simply the mean and covariance matrix of the conditional distribution of Y given X. Note that

Var(Y |X)= E{[Y − E(Y |X)][Y − E(Y |X)]′|X}
= E{YY′− 2Y [E(Y |X)]′+ [E(Y |X)][E(Y |X)]′|X}
= E{YY′|X} − E{2Y [E(Y |X)]′|X} + E{[E(Y |X)][E(Y |X)]′|X}
= E(YY′|X) − 2E{Y |X}[E(Y |X)]′+ E{[E(Y |X)][E(Y |X)]′|X}
= E(YY′|X) − [E(Y |X)][E(Y |X)]′

Following the proof of Theorem 4.4.7, we can also establish that Var(Y ) = E[Var(Y |X)] + Var(E(Y |X)) for random vectors X and Y .

beamer-tu-logo

Best prediction

Let X be a random variable on with E(X^{2}) < ∞ We want to predict the future value of X by constructing a g(Y ), where Y is another random variable currently observed. We now show that E(X|Y ) is the best predictor of X in the sense that

E[X − E(X|Y )]^{2} = min_{g:E[g(Y )]^{2}<∞} E[X − g(Y )]^{2}.

E[X − g(Y )]^{2} = E[X − E(X|Y ) + E(X|Y ) − g(Y )]^{2}
= E[X − E(X|Y )]^{2} + E[E(X|Y ) − g(Y )]^{2}
+2E{[X − E(X|Y )][E(X|Y ) − g(Y )]}
= E[X − E(X|Y )]^{2} + E[E(X|Y ) − g(Y )]^{2}
+2E{E{[X − E(X|Y )][E(X|Y ) − g(Y )]|Y }}
= E[X − E(X|Y )]^{2} + E[E(X|Y ) − g(Y )]^{2}
+2E{[E(X|Y ) − g(Y )]E[X − E(X|Y )|Y ]}
= E[X − E(X|Y )]^{2} + E[E(X|Y ) − g(Y )]^{2}
≥ E[X − E(X|Y )]^{2}
UW-Madison (Statistics)  Stat 609 Lecture 16  2014  11 / 11