beamer-tu-logo

Chapter 1: Probability Theory Lecture 1: Sample space and probability

Random experiment

An experiment (in a general sense) that results in one of more than one outcomes with uncertainty in which outcome will be the realization

Examples of random experiments

Tomorrow’s weather: Sunny, Cloudy, Rainy Tomorrow’s temperature: (−M,M) Number of car accidents in a city: 0, 1, 2, 3, ... The life time of a television set: (0,∞) Toss a coin: Heads or Tails?

Statistical inference

Treat data from the experiment as a sample from some population Make inference about the population Find things having regular patterns, not appear by chance

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  1 / 15

beamer-tu-logo

Chapter 1: Probability Theory Lecture 1: Sample space and probability

Random experiment

An experiment (in a general sense) that results in one of more than one outcomes with uncertainty in which outcome will be the realization

Examples of random experiments

Tomorrow’s weather: Sunny, Cloudy, Rainy Tomorrow’s temperature: (−M,M) Number of car accidents in a city: 0, 1, 2, 3, ... The life time of a television set: (0,∞) Toss a coin: Heads or Tails?

Statistical inference

Treat data from the experiment as a sample from some population Make inference about the population Find things having regular patterns, not appear by chance

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  1 / 15

beamer-tu-logo

Chapter 1: Probability Theory Lecture 1: Sample space and probability

Random experiment

An experiment (in a general sense) that results in one of more than one outcomes with uncertainty in which outcome will be the realization

Examples of random experiments

Tomorrow’s weather: Sunny, Cloudy, Rainy Tomorrow’s temperature: (−M,M) Number of car accidents in a city: 0, 1, 2, 3, ... The life time of a television set: (0,∞) Toss a coin: Heads or Tails?

Statistical inference

Treat data from the experiment as a sample from some population Make inference about the population Find things having regular patterns, not appear by chance

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  1 / 15

beamer-tu-logo

Probability theory

A basic tool for statistics, which explains a random experiment in terms of frequencies and their characteristics

For example, some past observations (data) may tell us that tomorrow’s weather is sunny with 80%, cloudy with 15%, and rainy with 5%.

To study statistical inference, we must have a very good knowledge in probability theory.

Definition 1.1.1. Sample space (or outcome space)

The set S containing all possible outcomes is called the sample space

Example

S = {Heads,Tails}, = {0,1}, = {0,1,2,3,...}, = (−M,M), = R (all real numbers), = (0,∞)

Discrete sample space: S contains integers (finitely or infinitely many)

Continuous sample space: e.g., S = [0,1], R, or R^{2}.

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  2 / 15

beamer-tu-logo

Probability theory

A basic tool for statistics, which explains a random experiment in terms of frequencies and their characteristics

For example, some past observations (data) may tell us that tomorrow’s weather is sunny with 80%, cloudy with 15%, and rainy with 5%.

To study statistical inference, we must have a very good knowledge in probability theory.

Definition 1.1.1. Sample space (or outcome space)

The set S containing all possible outcomes is called the sample space

Example

S = {Heads,Tails}, = {0,1}, = {0,1,2,3,...}, = (−M,M), = R (all real numbers), = (0,∞)

Discrete sample space: S contains integers (finitely or infinitely many)

Continuous sample space: e.g., S = [0,1], R, or R^{2}.

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  2 / 15

beamer-tu-logo

Subsets or sets

In probability and statistics, we are interested in subsets of a given sample space S (sometime called sets for simplicity).

A ⊂ S: A is a subset of S.

Example: the life time of a television set is between 0 and 100 is a subset A = (0,100] of the sample space S = (0,∞).

Two very special subsets:

Empty set: ∅ is the subset of S containing no element.

The sample space S itself is a subset of S (the notation A ⊂ S allows the special case of A = S).

Expressions of subsets

A subset is expressed in words or math formulas inside of { }.

Example: A = {x : x satisfies some condition } A = {x : x ∈ R and x ≥ 0} = [0,∞)

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  3 / 15

beamer-tu-logo

Subsets or sets

In probability and statistics, we are interested in subsets of a given sample space S (sometime called sets for simplicity).

A ⊂ S: A is a subset of S.

Example: the life time of a television set is between 0 and 100 is a subset A = (0,100] of the sample space S = (0,∞).

Two very special subsets:

Empty set: ∅ is the subset of S containing no element.

The sample space S itself is a subset of S (the notation A ⊂ S allows the special case of A = S).

Expressions of subsets

A subset is expressed in words or math formulas inside of { }.

Example: A = {x : x satisfies some condition } A = {x : x ∈ R and x ≥ 0} = [0,∞)

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  3 / 15

beamer-tu-logo

Relationship

A ⊂ B: x ∈ A ⇒ x ∈ B A = B: A ⊂ B and B ⊂ A Disjoint: A and B have no common elements

Example

A = {all even integers } B = [2,∞) Obviously, for any x ∈ A, x ≥ 2 and thus x ∈ B. Hence A ⊂ B.

A more complicated example

Let

A = {x ∈ R^{k} : x′x ≤ c}

where c > 0 is a given constant, x′is the transpose of x, and a′b is the usual vector product, and

B = {x ∈ R^{k} : (y′x)^{2} ≤ cy′y for all y ∈ R^{k} }

We want to show A = B.

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  4 / 15

beamer-tu-logo

Relationship

A ⊂ B: x ∈ A ⇒ x ∈ B A = B: A ⊂ B and B ⊂ A Disjoint: A and B have no common elements

Example

A = {all even integers } B = [2,∞) Obviously, for any x ∈ A, x ≥ 2 and thus x ∈ B. Hence A ⊂ B.

A more complicated example

Let

A = {x ∈ R^{k} : x′x ≤ c}

where c > 0 is a given constant, x′is the transpose of x, and a′b is the usual vector product, and

B = {x ∈ R^{k} : (y′x)^{2} ≤ cy′y for all y ∈ R^{k} }

We want to show A = B.

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  4 / 15

beamer-tu-logo

Relationship

A ⊂ B: x ∈ A ⇒ x ∈ B A = B: A ⊂ B and B ⊂ A Disjoint: A and B have no common elements

Example

A = {all even integers } B = [2,∞) Obviously, for any x ∈ A, x ≥ 2 and thus x ∈ B. Hence A ⊂ B.

A more complicated example

Let

A = {x ∈ R^{k} : x′x ≤ c}

where c > 0 is a given constant, x′is the transpose of x, and a′b is the usual vector product, and

B = {x ∈ R^{k} : (y′x)^{2} ≤ cy′y for all y ∈ R^{k} }

We want to show A = B.

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  4 / 15

beamer-tu-logo

A more complicated example

First, we show A ⊂ B. For any x ∈ A, x′x ≤ c and by the Cauchy-Schwartz inequality,

(y′x)^{2} ≤ x′xy′y ≤ c(y′y)

for all y ∈ R^{k}. This means x ∈ B and thus A ⊂ B.

Next, we show B ⊂ A. If x ∈ B,

(y′x)^{2} ≤ cy′y for all y ∈ R^{k}

In particular, y = x should satisfy this inequality, which means

(x′x)^{2} ≤ cx′x ⇒ x′x ≤ c

This means x ∈ A and thus B ⊂ A.

beamer-tu-logo

Set operation

The union of A and B: A ∪ B = {x: x ∈ A or x ∈ B} The intersection of A and B: A ∩ B = {x: x ∈ A and x ∈ B} Disjoint A and B: A ∩ B = ∅ (the empty set) The complement of A: A^{c} = {x : x ∉ A}, S^{c} = ∅ Difference of A and B: A − B = A ∩ B^{c}

Venn Diagram

beamer-tu-logo

Properties

A ∩ ∅ = ∅ A ∪ ∅ = A A ∩ S = A A ∪ S = S ∅ ⊂ A ∩ B ⊂ A or B ⊂ A ∪ B ⊂ S A ⊂ B ⇒ A ∪ B = B and A ∩ B = A A − B = A − A ∩ B A = (A ∩ B) ∪ (A ∩ B^{c}) A ∪ B = A ∪ (B − A) = A ∪ (B − A ∩ B) ∪_{i=1}^{∞}A_{i} = A_{1} ∪ A_{2} ∪ ···= {x : x ∈ A_{i} for some i } ∩_{i=1}^{∞}A_{i} = A_{1} ∩ A_{2} ∩ ···= {x : x ∈ A_{i} for all i } B ∩ (∪_{i=1}^{∞}A_{i}) = ∪_{i=1}^{∞}(B ∩ A_{i}) B ∪ (∩_{i=1}^{∞}A_{i}) = ∩_{i=1}^{∞}(B ∪ A_{i}) DeMorgan’s laws: (∪_{i=1}^{∞}A_{i})^{c} = ∩_{i=1}^{∞}A_{i}^{c} and (∩_{i=1}^{∞}A_{i})^{c} = ∪_{i=1}^{∞}A_{i}^{c}

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  7 / 15

beamer-tu-logo

Subset occurrence

If x in a subset A is the realization of an experiment, then we say that A occurs. Let A, B, C be 3 subsets. Their occurrence can be expressed by the following subsets:

A occurs: A All subsets occur: A ∩ B ∩ C At least one subset occurs: A ∪ B ∪ C Only A occurs: A ∩ B^{c} ∩ C^{c} Exactly one subset occurs: (A ∩ B^{c} ∩ C^{c}) ∪ (A^{c} ∩ B ∩ C^{c}) ∪ (A^{c} ∩ B^{c} ∩ C) None occurs: A^{c} ∩ B^{c} ∩ C^{c} Not all subsets occur: A^{c} ∪ B^{c} ∪ C^{c} At least two subsets occur: (A ∩ B) ∪ (A ∩ C) ∪ (B ∩ C) At most one subset occurs: (A ∩ B^{c} ∩ C^{c}) ∪ (A^{c} ∩ B ∩ C^{c}) ∪ (A^{c} ∩ B^{c} ∩ C) ∪ (A^{c} ∩ B^{c} ∩ C^{c})

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  8 / 15

beamer-tu-logo

Definition 1.2.1

A collection F of subsets of a sample space S is called a σ-field (or σ-algebra) iff it has the following properties:

(i) The empty set ∅ ∈ F; (ii) If A ∈ F, then the complement A^{c} ∈ F; (iii) If A_{i} ∈ F, i = 1,2,..., then their union ∪A_{i} ∈ F.

If A ∈ F, then A is called an event. Why do we need to consider σ-fields? When S is continuous, there are some “bad" subsets for which a logical definition of measure or probability is not possible. F contains good subsets we deal with, but they have to satisfy (i)-(iii). If S is discrete, we do not need this concept, i.e., F = all subsets of S. In R^{k}, the k-dimensional Euclidean space (R^{1} = R is the real line), we consider F to be the smallest σ-field containing all open sets, and sets in F are called Borel sets.

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  9 / 15

beamer-tu-logo

Definition 1.2.1

A collection F of subsets of a sample space S is called a σ-field (or σ-algebra) iff it has the following properties:

(i) The empty set ∅ ∈ F; (ii) If A ∈ F, then the complement A^{c} ∈ F; (iii) If A_{i} ∈ F, i = 1,2,..., then their union ∪A_{i} ∈ F.

If A ∈ F, then A is called an event. Why do we need to consider σ-fields? When S is continuous, there are some “bad" subsets for which a logical definition of measure or probability is not possible. F contains good subsets we deal with, but they have to satisfy (i)-(iii). If S is discrete, we do not need this concept, i.e., F = all subsets of S. In R^{k}, the k-dimensional Euclidean space (R^{1} = R is the real line), we consider F to be the smallest σ-field containing all open sets, and sets in F are called Borel sets.

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  9 / 15

beamer-tu-logo

How to define the probability of an event A?

Intuitive definition: the rate or percentage of the occurrence of A Statistical definition: if we repeat the same experiment M times, the probability of A =(the number of times A occurs)/M when M is sufficiently large. Mathematical definition: probability is a special measure. A measure is an abstract extension of length, area, volume, ...

Definition 1.2.4

A set function ν defined on a σ-field F is called a measure iff it has the following properties.

(i) 0 ≤ ν(A) ≤ ∞ for any A ∈ F. (ii) ν(∅) = 0. (iii) If A_{i} ∈ F, i = 1,2,..., and A_{i}’s are disjoint, i.e., A_{i} ∩ A_{j} = ∅ for any i ≠ j, then

ν ( ⋃_{i=1}^{∞}A_{i}) = ∑_{i=1}^{∞}ν(A_{i}).

beamer-tu-logo

How to define the probability of an event A?

Intuitive definition: the rate or percentage of the occurrence of A Statistical definition: if we repeat the same experiment M times, the probability of A =(the number of times A occurs)/M when M is sufficiently large. Mathematical definition: probability is a special measure. A measure is an abstract extension of length, area, volume, ...

Definition 1.2.4

A set function ν defined on a σ-field F is called a measure iff it has the following properties.

(i) 0 ≤ ν(A) ≤ ∞ for any A ∈ F. (ii) ν(∅) = 0. (iii) If A_{i} ∈ F, i = 1,2,..., and A_{i}’s are disjoint, i.e., A_{i} ∩ A_{j} = ∅ for any i ≠ j, then

ν ( ⋃_{i=1}^{∞}A_{i}) = ∑_{i=1}^{∞}ν(A_{i}).

beamer-tu-logo

Theorem 1.2.6.

Let S = {s_{1},s_{2},...} and F be all subsets of S. Let p_{1},p_{2},... be nonnegative numbers that sum to 1. For any A ∈ F, define

P(A) = ∑_{i:s_{i}∈A} p_{i}

(sum over the empty set is defined to be 0). Then P is a probability on F.

Theorem 1.2.8.

If P is a probability and A is an event, then a. P(∅) = 0; b. P(A) ≤ 1; c. P(A^{c}) = 1 − P(A).

Theorem 1.2.9.

If P is a probability and A and B are events, then a. P(B ∩ A^{c}) = P(B) − P(A ∩ B); b. P(A ∪ B) = P(A) + P(B) − P(A ∩ B); c. If A ⊂ B, then P(A) ≤ P(B).

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  11 / 15

beamer-tu-logo

Theorem 1.2.6.

Let S = {s_{1},s_{2},...} and F be all subsets of S. Let p_{1},p_{2},... be nonnegative numbers that sum to 1. For any A ∈ F, define

P(A) = ∑_{i:s_{i}∈A} p_{i}

(sum over the empty set is defined to be 0). Then P is a probability on F.

Theorem 1.2.8.

If P is a probability and A is an event, then a. P(∅) = 0; b. P(A) ≤ 1; c. P(A^{c}) = 1 − P(A).

Theorem 1.2.9.

If P is a probability and A and B are events, then a. P(B ∩ A^{c}) = P(B) − P(A ∩ B); b. P(A ∪ B) = P(A) + P(B) − P(A ∩ B); c. If A ⊂ B, then P(A) ≤ P(B).

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  11 / 15

beamer-tu-logo

Theorem 1.2.6.

Let S = {s_{1},s_{2},...} and F be all subsets of S. Let p_{1},p_{2},... be nonnegative numbers that sum to 1. For any A ∈ F, define

P(A) = ∑_{i:s_{i}∈A} p_{i}

(sum over the empty set is defined to be 0). Then P is a probability on F.

Theorem 1.2.8.

If P is a probability and A is an event, then a. P(∅) = 0; b. P(A) ≤ 1; c. P(A^{c}) = 1 − P(A).

Theorem 1.2.9.

If P is a probability and A and B are events, then a. P(B ∩ A^{c}) = P(B) − P(A ∩ B); b. P(A ∪ B) = P(A) + P(B) − P(A ∩ B); c. If A ⊂ B, then P(A) ≤ P(B).

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  11 / 15

beamer-tu-logo

Example.

Select randomly 4 cards from a deck 52 cards. Let A be the event that 1 ace and 1 king are selected and B be the event that 2 aces are selected.

P(A) = \begin{matrix}\binom{4}{1} & \binom{4}{1} & \binom{44}{2} \\ ( & {52 \atop 4} & ) \end{matrix} = \frac{4 × 4 × 44 × 43/2}{270725} = 0.055909

P(B) = \begin{matrix}\binom{4}{2} & \binom{48}{2} \\ ( & {52 \atop 4} & ) \end{matrix} = \frac{6 × 48 × 47/2}{270725} = 0.025000

P(A^{c}) = 1 − P(A) = 1 − 0.055909 = 0.944091

P(A ∩ B) = \begin{matrix}\binom{4}{1} & \binom{4}{2} & \binom{44}{1} \\ ( & {52 \atop 4} & ) \end{matrix} = \frac{4 × 6 × 44}{270725} = 0.003901

P(A − B) = P(A) − P(A ∩ B) = 0.055909 − 0.003901 = 0.052008

P(A ∪ B) = P(A) + P(B) − P(A ∩ B) = 0.052008 + 0.025 = 0.077008

beamer-tu-logo

Birth date problem

There are n balls in N boxes, n ≤ N. What is the probability that there are n boxes having exactly one ball?

P = \frac{N(N − 1)··· (N − n + 1)}{N^{n}} = \frac{N!}{N^{n}(N − n)!}

When there are n persons in a party, what is the probability p_{n} that there are at least two persons having the same birth date? n persons = n balls N = 365 days

P(no one has the same birth date with others ) = \frac{365!}{365^{n}(365 − n)!}

p_{n} = 1 − \frac{365!}{365^{n}(365 − n)!}

p_{20} = 0.4058, p_{30} = 0.6963, p_{50} = 0.9651, p_{60} = 0.9922

Reading assignments: Sections 1.2.3 and 1.2.4

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  13 / 15

beamer-tu-logo

Birth date problem

There are n balls in N boxes, n ≤ N. What is the probability that there are n boxes having exactly one ball?

P = \frac{N(N − 1)··· (N − n + 1)}{N^{n}} = \frac{N!}{N^{n}(N − n)!}

When there are n persons in a party, what is the probability p_{n} that there are at least two persons having the same birth date? n persons = n balls N = 365 days

P(no one has the same birth date with others ) = \frac{365!}{365^{n}(365 − n)!}

p_{n} = 1 − \frac{365!}{365^{n}(365 − n)!}

p_{20} = 0.4058, p_{30} = 0.6963, p_{50} = 0.9651, p_{60} = 0.9922

Reading assignments: Sections 1.2.3 and 1.2.4

UW-Madison (Statistics)  Stat 609 Lecture 1  2014  13 / 15

beamer-tu-logo

Union of three events

P(A ∪ B ∪ C) = P(A) + P(B) + P(C)

−P(A ∩ B) − P(A ∩ C) − P(B ∩ C) + P(A ∩ B ∩ C)

Example: P(A) = P(B) = P(C) = 1/4, A and B are disjoint, P(B ∩ C) = P(A ∩ C) = 1/6 P(A ∪ B ∪ C) =?

P(A ∪ B ∪ C) = 1/4 + 1/4 + 1/4 − 0 − 1/6 − 1/6 + 0 = 7/12

General addition formula

For events A_{1},...,A_{n},

P ( ⋃_{i=1}^{n}A_{i}) = ∑_{i=1}^{n}P(A_{i}) − ∑_{i<j} P(A_{i} ∩ A_{j})

+ ∑_{i<j<k} P(A_{i} ∩ A_{j}∩ A_{k}) + ··· + (−1)^{n−1} P(A_{1} ∩ A_{2} ∩ ··· ∩ A_{n})

beamer-tu-logo

Union of three events

P(A ∪ B ∪ C) = P(A) + P(B) + P(C)

−P(A ∩ B) − P(A ∩ C) − P(B ∩ C) + P(A ∩ B ∩ C)

Example: P(A) = P(B) = P(C) = 1/4, A and B are disjoint, P(B ∩ C) = P(A ∩ C) = 1/6 P(A ∪ B ∪ C) =?

P(A ∪ B ∪ C) = 1/4 + 1/4 + 1/4 − 0 − 1/6 − 1/6 + 0 = 7/12

General addition formula

For events A_{1},...,A_{n},

P ( ⋃_{i=1}^{n}A_{i}) = ∑_{i=1}^{n}P(A_{i}) − ∑_{i<j} P(A_{i} ∩ A_{j})

+ ∑_{i<j<k} P(A_{i} ∩ A_{j}∩ A_{k}) + ··· + (−1)^{n−1} P(A_{1} ∩ A_{2} ∩ ··· ∩ A_{n})

beamer-tu-logo

Theorem 1.2.11.

If P is a probability, A, A_{1},A_{2},..., C_{1},C_{2},... are events, and C_{i} ∩ C_{j} = ∅ for any i ≠ j and ∪_{i=1}^{∞}C_{i} = S, then a. P(A) = ∑ _{i=1}^{∞}P(A ∩ C_{i}); b. P(∪_{i=1}^{∞}A_{i}) ≤ ∑ _{i=1}^{∞}P(A_{i}).

Property a is useful when P(A ∩ C_{i}) is easy to calculate.

Property b is called Boole’s inequality.

Bonferroni’s inequality

For n events A_{1},..,A_{n},

1 − P ( ⋂_{i=1}^{n}A_{i}) = P ( ⋃_{i=1}^{n}A_{i}^{c}) ≤ ∑_{i=1}^{n}P(A_{i}^{c}) = n − ∑_{i=1}^{n}P(A_{i})

Then

P ( ⋂_{i=1}^{n}A_{i}) ≥ ∑_{i=1}^{n}P(A_{i}) − (n − 1)

beamer-tu-logo

Theorem 1.2.11.

If P is a probability, A, A_{1},A_{2},..., C_{1},C_{2},... are events, and C_{i} ∩ C_{j} = ∅ for any i ≠ j and ∪_{i=1}^{∞}C_{i} = S, then a. P(A) = ∑ _{i=1}^{∞}P(A ∩ C_{i}); b. P(∪_{i=1}^{∞}A_{i}) ≤ ∑ _{i=1}^{∞}P(A_{i}).

Property a is useful when P(A ∩ C_{i}) is easy to calculate.

Property b is called Boole’s inequality.

Bonferroni’s inequality

For n events A_{1},..,A_{n},

1 − P ( ⋂_{i=1}^{n}A_{i}) = P ( ⋃_{i=1}^{n}A_{i}^{c}) ≤ ∑_{i=1}^{n}P(A_{i}^{c}) = n − ∑_{i=1}^{n}P(A_{i})

Then

P ( ⋂_{i=1}^{n}A_{i}) ≥ ∑_{i=1}^{n}P(A_{i}) − (n − 1)