beamer-tu-logo

Lecture 2: Conditional probability and independence

Conditional probability

Sometimes we want to know the probability of event A given that another event B has occurred.

Definition 1.3.2.

If A and B are events with P(B) > 0, then the conditional probability of A given B is

P(A|B) = \frac{P(A ∩ B)}{P(B)}.
UW-Madison (Statistics)  Stat 609 Lecture 2  2014  1 / 13

beamer-tu-logo

The conditional probability can be viewed as:

The sample space is reduced to B. Then A occurs iff A ∩ B occurs. To have a probability, we need to divide P(A ∩ B) by P(B).

If A ∩ B = ∅, then P(A ∩ B) = 0 and P(A|B) = 0.

For convenience, we define P(A|B) = 0 when P(B) = 0.

P(A ∩ B) = P(A|B)P(B) = P(B|A)P(A).

P(·|B) is a probability:

P(∅|B) = 0, P(S|B) = P(B|B) = 1;

P(A^{c}|B) = 1 − P(A|B);

P(A ∪ C|B) = P(A|B) + P(C|B) − P(A ∩ C|B).

Example 1.3.4 (Three prisoners)

Three prisoners, A, B, and C, are on death row. The warden knows that one of the three prisoners will be pardoned. Prisoner A tries to get the warden to tell him who had been pardoned, but the warden refuses. Prisoner A then asks which of B or C will be executed. The warden thinks for a while, then tells A that B is to be executed.

UW-Madison (Statistics)  Stat 609 Lecture 2  2014  2 / 13

beamer-tu-logo

Prisoner A’s thinking:

Let A, B, and C denote the events that A, B, and C is pardoned, respectively. Then P(A) = P(B) = P(C) = 1/3. Given that B will be executed, then either A or C will be pardoned; my chance of being pardoned has risen from 0.33 to 0.5. Prisoner A’s calculation: P(A|B^{c}) = \frac{1}{2}.

Warden’s reasoning: Each prisoner has a 1/3 chance of being pardoned; either B or C must be executed, so I have given A no information about whether A will be pardoned.

Prisoner pardoned Warden tells A Probability
A  B dies  1/6
A  C dies  1/6
B  C dies  1/3
C  B dies  1/3

P(A|warden says B dies ) = \frac{P(warden says B dies ∩ A)}{P(wanden says B dies )} = \frac{1/6}{1/6 + 1/3} = \frac{1}{3}

Reason for the difference: {warden says B dies } ⊂ B^{c} but the two are not the same; e.g., the warden will not tell A whether A dies or not.

UW-Madison (Statistics)  Stat 609 Lecture 2  2014  3 / 13

beamer-tu-logo

Useful formulas

For events A_{1}, A_{2},...,

P ( ⋂_{i=1}^{n}A_{i}) = P(A_{1})P(A_{2}|A_{1})P(A_{3}|A_{1} ∩ A_{2})··· P ( A_{n}| ⋂_{i=1}^{n−1}A_{i})

For disjoint B_{1}, B_{2},..., with ∪_{i=1}^{∞}B_{i} = S,

P(A) = ∑_{i=1}^{∞}P(B_{i})P(A|B_{i})

In particular, P(A) = P(B)P(A|B) + P(B^{c})P(A|B^{c}).

Example

There are 90 red balls and 10 white balls. We select randomly one ball at a time without replacement. What is the probability that the 1st white ball appears at the 3rd draw? Define A_{i} = {the ith selection is a white ball} B_{i} = {the ith selection is a red ball}

P(B_{1} ∩ B_{2} ∩ A_{3}) = P(B_{1})P(B_{2}|B_{1})P(A_{3}|B_{1} ∩ B_{2}) = \frac{90}{100} × \frac{89}{99} × \frac{10}{98}

There are a white balls and b black balls in bag I, and c white balls and d black balls in bag II. Randomly select one ball from bag I and put it in bag II. Randomly select one ball from bag II. What is the probability that the 2nd selected ball is white? A = {the 2nd selection is a white ball} B = {the 1st selection is a white ball}

UW-Madison (Statistics)  Stat 609 Lecture 2  2014  4 / 13

beamer-tu-logo

Useful formulas

For events A_{1}, A_{2},...,

P ( ⋂_{i=1}^{n}A_{i}) = P(A_{1})P(A_{2}|A_{1})P(A_{3}|A_{1} ∩ A_{2})··· P ( A_{n}| ⋂_{i=1}^{n−1}A_{i})

For disjoint B_{1}, B_{2},..., with ∪_{i=1}^{∞}B_{i} = S,

P(A) = ∑_{i=1}^{∞}P(B_{i})P(A|B_{i})

In particular, P(A) = P(B)P(A|B) + P(B^{c})P(A|B^{c}).

Example

There are 90 red balls and 10 white balls. We select randomly one ball at a time without replacement. What is the probability that the 1st white ball appears at the 3rd draw? Define A_{i} = {the ith selection is a white ball} B_{i} = {the ith selection is a red ball}

P(B_{1} ∩ B_{2} ∩ A_{3}) = P(B_{1})P(B_{2}|B_{1})P(A_{3}|B_{1} ∩ B_{2}) = \frac{90}{100} × \frac{89}{99} × \frac{10}{98}

There are a white balls and b black balls in bag I, and c white balls and d black balls in bag II. Randomly select one ball from bag I and put it in bag II. Randomly select one ball from bag II. What is the probability that the 2nd selected ball is white? A = {the 2nd selection is a white ball} B = {the 1st selection is a white ball}

UW-Madison (Statistics)  Stat 609 Lecture 2  2014  4 / 13

beamer-tu-logo

Sensitive questions

People may not give you a true answer if he/she is asked “did you cheat in the last exam"? If we want to know the probability that a person cheats in a given exam, we may use the following procedure. We ask a person to flip a fair coin. If the result is “heads", then the question is “did you cheat"? If the result is “tails", then the question is “is your birthday afetr July 1"? No one, except the person being asked, knows which question is asked. The person will give the truthful answer.

A = the event that the person answers “yes". B = the event that the person is asked the 1st question.

P(A) = P(B)P(A|B) + P(B^{c})P(A|B^{c}) = 0.5 × P(A|B) + 0.5 × 0.5
P(A|B) = 2P(A) − 0.5

If we estimate P(A) by asking 10,000 people, then we can estimate P(A|B).

UW-Madison (Statistics)  Stat 609 Lecture 2  2014  5 / 13

beamer-tu-logo

Sensitive questions

People may not give you a true answer if he/she is asked “did you cheat in the last exam"? If we want to know the probability that a person cheats in a given exam, we may use the following procedure. We ask a person to flip a fair coin. If the result is “heads", then the question is “did you cheat"? If the result is “tails", then the question is “is your birthday afetr July 1"? No one, except the person being asked, knows which question is asked. The person will give the truthful answer.

A = the event that the person answers “yes". B = the event that the person is asked the 1st question.

P(A) = P(B)P(A|B) + P(B^{c})P(A|B^{c}) = 0.5 × P(A|B) + 0.5 × 0.5
P(A|B) = 2P(A) − 0.5

If we estimate P(A) by asking 10,000 people, then we can estimate P(A|B).

UW-Madison (Statistics)  Stat 609 Lecture 2  2014  5 / 13

beamer-tu-logo

Theorem 1.3.5. Bayes formula

Let B, A_{1},A_{2},... be events with A_{i} ∩ A_{j} = ∅, i ≠ j, and ∪_{i=1}^{∞}A_{i} = S. Then,

P(A_{i}|B) = \frac{P(B|A_{i})P(A_{i})}{∑_{j=1}^{∞}P(B|A_{j})P(A_{j})}.

In particular,

P(A|B) = \frac{P(B|A)P(A)}{P(B|A)P(A) + P(B|A^{c})P(A^{c})}

Example

A = the event that a person has a type of disease, P(A) = 0.0004. B = a medical test indicates that a tested person has the disease, P(B|A) = 0.99, P(B|A^{c}) = 0.05. P(A|B) =? By Bayes formula,

P(A|B) = \frac{0.0004 × 0.99}{0.0004 × 0.99 + 0.9996 × 0.05} = 0.007808

UW-Madison (Statistics)  Stat 609 Lecture 2  2014  6 / 13

beamer-tu-logo

Theorem 1.3.5. Bayes formula

Let B, A_{1},A_{2},... be events with A_{i} ∩ A_{j} = ∅, i ≠ j, and ∪_{i=1}^{∞}A_{i} = S. Then,

P(A_{i}|B) = \frac{P(B|A_{i})P(A_{i})}{∑_{j=1}^{∞}P(B|A_{j})P(A_{j})}.

In particular,

P(A|B) = \frac{P(B|A)P(A)}{P(B|A)P(A) + P(B|A^{c})P(A^{c})}

Example

A = the event that a person has a type of disease, P(A) = 0.0004. B = a medical test indicates that a tested person has the disease, P(B|A) = 0.99, P(B|A^{c}) = 0.05. P(A|B) =? By Bayes formula,

P(A|B) = \frac{0.0004 × 0.99}{0.0004 × 0.99 + 0.9996 × 0.05} = 0.007808

UW-Madison (Statistics)  Stat 609 Lecture 2  2014  6 / 13

beamer-tu-logo

Definition 1.3.7.

Two events A and B are (statistically) independent iff

P(A ∩ B) = P(A)P(B).

This definition is still good if P(A) = 0 or P(B) = 0. A and B are independent iff P(A|B) = P(A) (or P(B|A) = P(B)). That is, the probability of the occurrence of A is not affected by the occurrence of B. If A and B are independent, then the following pairs are also independent: A and B^{c}, A^{c} and B, and A^{c} and B^{c}; e.g.,

P(A ∩ B^{c}) = P(A) − P(A ∩ B) = P(A) − P(A)P(B) = P(A)P(B^{c})

Example 1.3.11 (Letters)

S consists of the 3! permutations of the letters a, b, and c, along with the three triplets of each letter:

S = \{ \begin{matrix}aaa & bbb & ccc \\ abc & bca & cba \\ acb & bac & cab \end{matrix} \}

beamer-tu-logo

Definition 1.3.7.

Two events A and B are (statistically) independent iff

P(A ∩ B) = P(A)P(B).

This definition is still good if P(A) = 0 or P(B) = 0. A and B are independent iff P(A|B) = P(A) (or P(B|A) = P(B)). That is, the probability of the occurrence of A is not affected by the occurrence of B. If A and B are independent, then the following pairs are also independent: A and B^{c}, A^{c} and B, and A^{c} and B^{c}; e.g.,

P(A ∩ B^{c}) = P(A) − P(A ∩ B) = P(A) − P(A)P(B) = P(A)P(B^{c})

Example 1.3.11 (Letters)

S consists of the 3! permutations of the letters a, b, and c, along with the three triplets of each letter:

S = \{ \begin{matrix}aaa & bbb & ccc \\ abc & bca & cba \\ acb & bac & cab \end{matrix} \}

beamer-tu-logo

Suppose that each element of S has probability 1/9 to occur. Define A_{i} = {ith place in the triple is occupied by a}, i = 1,2,3. By looking at S, we obtain that

P(A_{1}) = P(A_{2}) = P(A_{3}) = 1/3

P(A_{1} ∩ A_{2}) = P(A_{1} ∩ A_{3}) = P(A_{2} ∩ A_{3}) = 1/9

Thus,

P(A_{i} ∩ A_{j}) = P(A_{i})P(A_{j}) i ≠ j

and A_{i} and A_{j} are independent for any pair of (i,j), i ≠ j. It is easy to find events that are not independent, e.g., C = {abc,bca},

P(C) = 2/9, P(A_{1} ∩ C) = 1/9 ≠ P(C)P(A_{1}) = 2/27

If A and B has some relationship, usually they are not independent. If A ⊂ B, then P(A) = P(A ∩ B) = P(A)P(B) iff P(A) = 0 or P(B) = 1. If A ∩ B = ∅, then P(A ∩ B) = P(A)P(B) = 0 iff P(A) = 0 or P(B) = 0.

UW-Madison (Statistics)  Stat 609 Lecture 2  2014  8 / 13

beamer-tu-logo

Definition 1.3.12.

A collection of events A_{1},...,A_{n} are (mutually) independent iff for any subcollection A_{i_{1}},...,A_{i_{k}},

P (A_{i_{1}} ∩ ··· ∩ A_{i_{k}}) = P(A_{i_{1}})··· P(A_{i_{k}})

If we know P(A_{i}), i = 1,...,n, we may not know P(∪_{i=1}^{n}A_{i}). But if A_{1},...,A_{n} are independent, then

P ( ⋃_{i=1}^{n}A_{i}) = ∑_{i=1}^{n}P(A_{i}) − ∑_{i<j} P(A_{i})P(A_{j})

+ ∑_{i<j<k} P(A_{i})P(A_{j})P(A_{k}) + ··· + (−1)^{n−1} P(A_{1})P(A_{2})··· P(A_{n})

If events A, B, C are independent, then so are A ∪ B and C, A ∩ B and C, and A − B and C, e.g.,

P((A ∪ B) ∩ C) = P((A ∩ C) ∪ (B ∩ C))
= P(A ∩ C) + P(B ∩ C) − P(A ∩ C ∩ B)
= P(A)P(C) + P(B)P(C) − P(A)P(C)P(B)
= P(C)[P(A) + P(B) − P(A)P(B)] = P(C)P(A ∪ B)
UW-Madison (Statistics)  Stat 609 Lecture 2  2014  9 / 13

beamer-tu-logo

Definition 1.3.12.

A collection of events A_{1},...,A_{n} are (mutually) independent iff for any subcollection A_{i_{1}},...,A_{i_{k}},

P (A_{i_{1}} ∩ ··· ∩ A_{i_{k}}) = P(A_{i_{1}})··· P(A_{i_{k}})

If we know P(A_{i}), i = 1,...,n, we may not know P(∪_{i=1}^{n}A_{i}). But if A_{1},...,A_{n} are independent, then

P ( ⋃_{i=1}^{n}A_{i}) = ∑_{i=1}^{n}P(A_{i}) − ∑_{i<j} P(A_{i})P(A_{j})

+ ∑_{i<j<k} P(A_{i})P(A_{j})P(A_{k}) + ··· + (−1)^{n−1} P(A_{1})P(A_{2})··· P(A_{n})

If events A, B, C are independent, then so are A ∪ B and C, A ∩ B and C, and A − B and C, e.g.,

P((A ∪ B) ∩ C) = P((A ∩ C) ∪ (B ∩ C))
= P(A ∩ C) + P(B ∩ C) − P(A ∩ C ∩ B)
= P(A)P(C) + P(B)P(C) − P(A)P(C)P(B)
= P(C)[P(A) + P(B) − P(A)P(B)] = P(C)P(A ∪ B)
UW-Madison (Statistics)  Stat 609 Lecture 2  2014  9 / 13

beamer-tu-logo

Pairwise independence

Events A_{1},...,A_{n} are pairwise independent iff A_{i} and A_{j} are independent for any pair (i,j). Mutual independence is stronger than pairwise independence.

Example 1.3.11 (continued)

A_{1},A_{2},A_{3} are pairwise independent. They are not mutually independent, because

P(A_{1} ∩ A_{2} ∩ A_{3}) = \frac{1}{9} ≠ \frac{1}{27} = P(A_{1})P(A_{2})P(A_{3})

Example 1.3.10 (Tossing two dice)

An experiment consists of tossing two dice. The sample space is

S = \{ \begin{matrix}(1,1) & ··· & (1,6) \\ (2,1) & ··· & (2,6) \\ ··· & ··· & ··· \\ (6,1) & ··· & (6,6) \end{matrix} \}

beamer-tu-logo

Pairwise independence

Events A_{1},...,A_{n} are pairwise independent iff A_{i} and A_{j} are independent for any pair (i,j). Mutual independence is stronger than pairwise independence.

Example 1.3.11 (continued)

A_{1},A_{2},A_{3} are pairwise independent. They are not mutually independent, because

P(A_{1} ∩ A_{2} ∩ A_{3}) = \frac{1}{9} ≠ \frac{1}{27} = P(A_{1})P(A_{2})P(A_{3})

Example 1.3.10 (Tossing two dice)

An experiment consists of tossing two dice. The sample space is

S = \{ \begin{matrix}(1,1) & ··· & (1,6) \\ (2,1) & ··· & (2,6) \\ ··· & ··· & ··· \\ (6,1) & ··· & (6,6) \end{matrix} \}

beamer-tu-logo

Consider

A = {the same digit appear twice }  P(A) = 1/6
B = {the sum is between 7 and 10 }  P(B) = 1/2
C = {the sum is 2, or 7, or 8 }  P(C) = 1/3

Then

P(A ∩ B ∩ C) = P(\{(4,4)\}) = \frac{1}{36} = \frac{1}{6} × \frac{1}{2} × \frac{1}{3} = P(A)P(B)P(C)

However, A, B, and C are not independent, because

P(B ∩ C) = P(\{the sum is 7 or 8 \}) = \frac{11}{36} ≠ \frac{1}{6} = P(B)P(C)

Independent experiments

Two experiments are independent iff any event in experiment 1 and any event in experiment 2 are independent. n experiments are independent iff A_{1},...,A_{n} are independent, where A_{j} is any event from experiment j, j = 1,...,n. Independence of experiments is an important concept in statistics.

UW-Madison (Statistics)  Stat 609 Lecture 2  2014  11 / 13

beamer-tu-logo

Consider

A = {the same digit appear twice }  P(A) = 1/6
B = {the sum is between 7 and 10 }  P(B) = 1/2
C = {the sum is 2, or 7, or 8 }  P(C) = 1/3

Then

P(A ∩ B ∩ C) = P(\{(4,4)\}) = \frac{1}{36} = \frac{1}{6} × \frac{1}{2} × \frac{1}{3} = P(A)P(B)P(C)

However, A, B, and C are not independent, because

P(B ∩ C) = P(\{the sum is 7 or 8 \}) = \frac{11}{36} ≠ \frac{1}{6} = P(B)P(C)

Independent experiments

Two experiments are independent iff any event in experiment 1 and any event in experiment 2 are independent. n experiments are independent iff A_{1},...,A_{n} are independent, where A_{j} is any event from experiment j, j = 1,...,n. Independence of experiments is an important concept in statistics.

UW-Madison (Statistics)  Stat 609 Lecture 2  2014  11 / 13

beamer-tu-logo

Conditional independence

Events A and B are conditionally independent given event C iff

P(A ∩ B|C) = P(A|C)P(B|C)

Conditional independence is an important concept in statistics.

Independence does not imply conditional independence

In Example 1.3.11,

A_{i} = {ith place in the triple is occupied by a}, i = 1,2,3. A_{1} and A_{2} are independent, but they are not conditionally independent given A_{3}, because

P(A_{1} ∩ A_{2}|A_{3}) = \frac{P(A_{1} ∩ A_{2} ∩ A_{3})}{P(A_{3})} = \frac{1/9}{1/3} = \frac{1}{3}

P(A_{1}|A_{3})P(A_{2}|A_{3}) = \frac{P(A_{1} ∩ A_{3})}{P(A_{3})} \frac{P(A_{2} ∩ A_{3})}{P(A_{3})} = P(A_{1})P(A_{2}) = \frac{1}{9}

UW-Madison (Statistics)  Stat 609 Lecture 2  2014  12 / 13

beamer-tu-logo

Conditional independence

Events A and B are conditionally independent given event C iff

P(A ∩ B|C) = P(A|C)P(B|C)

Conditional independence is an important concept in statistics.

Independence does not imply conditional independence

In Example 1.3.11,

A_{i} = {ith place in the triple is occupied by a}, i = 1,2,3. A_{1} and A_{2} are independent, but they are not conditionally independent given A_{3}, because

P(A_{1} ∩ A_{2}|A_{3}) = \frac{P(A_{1} ∩ A_{2} ∩ A_{3})}{P(A_{3})} = \frac{1/9}{1/3} = \frac{1}{3}

P(A_{1}|A_{3})P(A_{2}|A_{3}) = \frac{P(A_{1} ∩ A_{3})}{P(A_{3})} \frac{P(A_{2} ∩ A_{3})}{P(A_{3})} = P(A_{1})P(A_{2}) = \frac{1}{9}

UW-Madison (Statistics)  Stat 609 Lecture 2  2014  12 / 13

beamer-tu-logo

Conditional independence does not imply independence

S = \{ \begin{matrix}aaa & bbb & ccc & abd \\ abc & bca & cba & acd \\ acb & bac & cab & bcd \end{matrix} \}

A_{i} = {ith place in the triple is occupied by a}, i = 1,2,3. B = {d does not appear }

A_{1} and A_{2} are conditionally independent given B, because

P(A_{1} ∩ A_{2}|B) = \frac{1}{9} = \frac{1}{3} × \frac{1}{3} = P(A_{1}|B)P(A_{2}|B)

However, A_{1} and A_{2} are not independent, because

P(A_{1} ∩ A_{2}) = \frac{1}{9} ≠ \frac{5}{12} × \frac{3}{12} = P(A_{1})P(A_{2})

Mutual independence implies conditional independence

If A, B, C are mutually independent, then A and B are conditionally independent given C, because

P(A ∩ B|C) = \frac{P(A ∩ B ∩ C)}{P(C)} = P(A)P(B) = P(A|C)P(B|C)

UW-Madison (Statistics)  Stat 609 Lecture 2  2014  13 / 13

beamer-tu-logo

Conditional independence does not imply independence

S = \{ \begin{matrix}aaa & bbb & ccc & abd \\ abc & bca & cba & acd \\ acb & bac & cab & bcd \end{matrix} \}

A_{i} = {ith place in the triple is occupied by a}, i = 1,2,3. B = {d does not appear }

A_{1} and A_{2} are conditionally independent given B, because

P(A_{1} ∩ A_{2}|B) = \frac{1}{9} = \frac{1}{3} × \frac{1}{3} = P(A_{1}|B)P(A_{2}|B)

However, A_{1} and A_{2} are not independent, because

P(A_{1} ∩ A_{2}) = \frac{1}{9} ≠ \frac{5}{12} × \frac{3}{12} = P(A_{1})P(A_{2})

Mutual independence implies conditional independence

If A, B, C are mutually independent, then A and B are conditionally independent given C, because

P(A ∩ B|C) = \frac{P(A ∩ B ∩ C)}{P(C)} = P(A)P(B) = P(A|C)P(B|C)

UW-Madison (Statistics)  Stat 609 Lecture 2  2014  13 / 13