beamer-tu-logo

Lecture 5: Expectations

The expected value, also called the expectation or mean, of a random variable is its average value weighted by its probability distribution.

Definition 2.2.1.

The expected value or mean of a random variable g(X) is

E[g(X)] = \{ \begin{matrix}∫_{−∞}^{∞} g(x)f_{X}(x)dx & if X has pdf f_{X} \\ ∑_{x} g(x)f_{X}(x) & if X has pmf f_{X} \end{matrix}

provided that the integral or the sum exists (is finite); otherwise we say that the expected value of g(X) does not exist.

The expected value is a number that summarizes a typical, middle, or expected value of an observation of the random variable. If g(X)≥0, then E[g(X)] is always defined except that it may be ∞. For any g(X), its expected value exists iff E|g(X)| < ∞. The expectation is associated with the distribution of X, not with X.

UW-Madison (Statistics)  Stat 609 Lecture 5  2014  1 / 11

beamer-tu-logo

Lecture 5: Expectations

The expected value, also called the expectation or mean, of a random variable is its average value weighted by its probability distribution.

Definition 2.2.1.

The expected value or mean of a random variable g(X) is

E[g(X)] = \{ \begin{matrix}∫_{−∞}^{∞} g(x)f_{X}(x)dx & if X has pdf f_{X} \\ ∑_{x} g(x)f_{X}(x) & if X has pmf f_{X} \end{matrix}

provided that the integral or the sum exists (is finite); otherwise we say that the expected value of g(X) does not exist.

The expected value is a number that summarizes a typical, middle, or expected value of an observation of the random variable. If g(X)≥0, then E[g(X)] is always defined except that it may be ∞. For any g(X), its expected value exists iff E|g(X)| < ∞. The expectation is associated with the distribution of X, not with X.

UW-Madison (Statistics)  Stat 609 Lecture 5  2014  1 / 11

beamer-tu-logo

Example

X has distribution

\begin{matrix}x & −2 & −1 & 0 & 1 & 2 & 3 \\ f_{X}(x) & 0.1 & 0.2 & 0.1 & 0.2 & 0.3 & 0.1 \end{matrix}

E(X) = ∑ xf_{X}(x) = −2 × 0.1 − 0.2 + 0.2 + 2 × 0.3 + 3 × 0.1 = 0.7 E(X^{2}) = ∑ x^{2}f_{X}(x) = 4 × 0.1 + 0.2 + 0.2 + 4 × 0.3 + 9 × 0.1 = 2.9

Y = g(X) = X^{2} has distribution

y  0 1 4 9
f_{Y}(y) 0.1 0.4 0.4 0.1
E(Y ) = E(X^{2}) = ∑ yf_{Y}(y) = 0.4 + 4 × 0.4 + 9 × 0.1 = 2.9

Example 2.2.2.

Suppose X ≥ 0 has pdf f_{X}(x) = λ^{−1} e^{−x/λ} , where λ > 0 is a constant. Then

beamer-tu-logo

Example

X has distribution

\begin{matrix}x & −2 & −1 & 0 & 1 & 2 & 3 \\ f_{X}(x) & 0.1 & 0.2 & 0.1 & 0.2 & 0.3 & 0.1 \end{matrix}

E(X) = ∑ xf_{X}(x) = −2 × 0.1 − 0.2 + 0.2 + 2 × 0.3 + 3 × 0.1 = 0.7 E(X^{2}) = ∑ x^{2}f_{X}(x) = 4 × 0.1 + 0.2 + 0.2 + 4 × 0.3 + 9 × 0.1 = 2.9

Y = g(X) = X^{2} has distribution

y  0 1 4 9
f_{Y}(y) 0.1 0.4 0.4 0.1
E(Y ) = E(X^{2}) = ∑ yf_{Y}(y) = 0.4 + 4 × 0.4 + 9 × 0.1 = 2.9

Example 2.2.2.

Suppose X ≥ 0 has pdf f_{X}(x) = λ^{−1} e^{−x/λ} , where λ > 0 is a constant. Then

beamer-tu-logo

In some cases, the calculation of the expectation requires some derivation.

Example 2.2.3.

Let X be a discrete random variable with pmf

f_{X}(x) = \binom{n}{x} p^{x}(1 − p)^{n−x} , x = 0,1,...,n,

where n is a fixed integer and 0 < p < 1 is a fixed constant. For any n and p,

∑_{x=0}^{n} \binom{n}{x} p^{x}(1 − p)^{n−x} = 1.

By definition and the fact that x\binom{n}{x} = n\binom{n−1}{x−1}, we obtain

E(X) = ∑_{x=0}^{n}x\binom{n}{x} p^{x}(1 − p)^{n−x} = ∑_{x=1}^{n}x\binom{n}{x} p^{x}(1 − p)^{n−x}

= ∑_{x=1}^{n}n\binom{n − 1}{x − 1} p^{x}(1 − p)^{n−x} = np ∑_{y=0}^{n−1} \binom{n − 1}{y} p^{y}(1 − p)^{n−1−y}
= np
UW-Madison (Statistics)  Stat 609 Lecture 5  2014  3 / 11

beamer-tu-logo

Example 2.2.4 (Nonexistence of expectation)

Suppose that X has the following pdf:

f_{X}(x) = \frac{1}{π(1 + x^{2})}, x ∈ R

Note that

E|X| = ∫_{−∞}^{∞} \frac{|x|}{π(1 + x^{2})}dx = \frac{2}{π} ∫_{0}^{∞} \frac{x}{1 + x^{2}} dx
= \frac{2}{π} lim_{M→∞} ∫_{0}^{M} \frac{x}{1 + x^{2}} dx = \frac{2}{π} lim_{M→∞} ∫_{0}^{M} \frac{1}{1 + x^{2}} d(x^{2}/2)
= \frac{2}{π} lim_{M→∞} \frac{log(1 + x^{2})}{2} |_{0}^{M} = \frac{2}{π} lim_{M→∞} \frac{log(1 + M^{2})}{2} = ∞

Thus, neither X nor |X| has an expectation. We should not wrongly think

E(X) = ∫_{−∞}^{∞} \frac{x}{π(1 + x^{2})}dx = lim_{M→∞} ∫_{−M}^{M} \frac{x}{π(1 + x^{2})}dx = lim_{M→∞} 0 = 0

beamer-tu-logo

Linear operation

Taking expectation is a linear operation in the sense that E(c) = c for any constant c; E(aX) = aE(X) for any constant a and random variable X; E(X + Y ) = E(X) + E(Y ) for any random variables X and Y . These are special cases of the following theorem.

Theorem 2.2.5

Let X and Y be random variables whose expectations exist, and let a, b, and c be constants. a. E(aX + bY + c) = aE(X) + bE(Y ) + c. b. If X ≥ Y , then E(X) ≥ E(Y ).

Proof.

The proof of Theorem 2.2.5 is easy if X = g_{1}(Z) and Y = g_{2}(Z) for a random variable Z and some functions g_{1} and g_{2}, and Z has a pdf or pmf.

In general, the proof of property a is not simple (a topic in stat 709).

UW-Madison (Statistics)  Stat 609 Lecture 5  2014  5 / 11

beamer-tu-logo

Linear operation

Taking expectation is a linear operation in the sense that E(c) = c for any constant c; E(aX) = aE(X) for any constant a and random variable X; E(X + Y ) = E(X) + E(Y ) for any random variables X and Y . These are special cases of the following theorem.

Theorem 2.2.5

Let X and Y be random variables whose expectations exist, and let a, b, and c be constants. a. E(aX + bY + c) = aE(X) + bE(Y ) + c. b. If X ≥ Y , then E(X) ≥ E(Y ).

Proof.

The proof of Theorem 2.2.5 is easy if X = g_{1}(Z) and Y = g_{2}(Z) for a random variable Z and some functions g_{1} and g_{2}, and Z has a pdf or pmf.

In general, the proof of property a is not simple (a topic in stat 709).

UW-Madison (Statistics)  Stat 609 Lecture 5  2014  5 / 11

beamer-tu-logo

Linear operation

Taking expectation is a linear operation in the sense that E(c) = c for any constant c; E(aX) = aE(X) for any constant a and random variable X; E(X + Y ) = E(X) + E(Y ) for any random variables X and Y . These are special cases of the following theorem.

Theorem 2.2.5

Let X and Y be random variables whose expectations exist, and let a, b, and c be constants. a. E(aX + bY + c) = aE(X) + bE(Y ) + c. b. If X ≥ Y , then E(X) ≥ E(Y ).

Proof.

The proof of Theorem 2.2.5 is easy if X = g_{1}(Z) and Y = g_{2}(Z) for a random variable Z and some functions g_{1} and g_{2}, and Z has a pdf or pmf.

In general, the proof of property a is not simple (a topic in stat 709).

UW-Madison (Statistics)  Stat 609 Lecture 5  2014  5 / 11

beamer-tu-logo

Example 2.2.6 (Minimizing distance)

The mean of a random variable X is a good guess (predict) of X. Suppose that we measure the distance between X and a constant b by (X − b)^{2}. The closer b is to X, the smaller this quantity is. We want to find a value b that minimizes the average E(X − b)^{2}. If μ = E(X), then

E(X − μ)^{2} = min_{b∈R}E(X − b)^{2}

To show this, note that

E(X − b)^{2} = E[(X − μ) + (μ − b))]2
= E[(X − μ)^{2}] + E[(μ − b)^{2}] + E[2(X − μ)(μ − b)]
= E(X − μ)^{2} + (μ − b)^{2} + 2(μ − b)E(X − μ)
= E(X − μ)^{2} + (μ − b)^{2}
≥ E(X − μ)^{2}.

There is an alternative proof using E(X − b)^{2} = E(X^{2}) − 2bE(X) + b^{2}.

UW-Madison (Statistics)  Stat 609 Lecture 5  2014  6 / 11

beamer-tu-logo

Nonlinear functions

When calculating expectations of nonlinear functions of X, we can proceed in one of two ways.

Using the distribution of X, we can calculate

E[g(X)] = ∫_{−∞}^{∞} g(x)f_{X}(x)dx  or  ∑_{x} g(x)f_{X}(x)

We can find the pdf or pmf of Y = g(X), f_{Y}, and then use

E[g(X)] = E(Y ) = ∫_{−∞}^{∞} yf_{Y}(y)dy  or  ∑_{y} yf_{Y}(y)

Example 2.2.7.

Suppose that X has a uniform pdf on (0,1),

f_{X}(x) = \{ \begin{matrix}1 & 0 < x < 1 \\ 0 & otherwise \end{matrix}

Consider g(X) = −logX.

beamer-tu-logo

Nonlinear functions

When calculating expectations of nonlinear functions of X, we can proceed in one of two ways.

Using the distribution of X, we can calculate

E[g(X)] = ∫_{−∞}^{∞} g(x)f_{X}(x)dx  or  ∑_{x} g(x)f_{X}(x)

We can find the pdf or pmf of Y = g(X), f_{Y}, and then use

E[g(X)] = E(Y ) = ∫_{−∞}^{∞} yf_{Y}(y)dy  or  ∑_{y} yf_{Y}(y)

Example 2.2.7.

Suppose that X has a uniform pdf on (0,1),

f_{X}(x) = \{ \begin{matrix}1 & 0 < x < 1 \\ 0 & otherwise \end{matrix}

Consider g(X) = −logX.

beamer-tu-logo

On the other hand, we can use Theorem 2.1.5 to obtain the pdf of Y = −logX:

f_{Y}(y) = f_{X}(e^{−y} )|e^{−y} | = e^{−y} , y > 0

Then

E[g(X)] = E(Y ) = ∫_{−∞}^{∞} yf_{Y}(y)dy = ∫_{0}^{∞} ye^{−y} dy = 1

Example

Suppose that X has the standard normal pdf

f_{X}(x) = \frac{1}{\sqrt{2π}} e^{−x^{2}/2} , x ∈ R

Consider g(X) = X^{2}. By Example 2.1.9, Y = X^{2} has the chi-square pdf

f_{Y}(y) = \frac{1}{\sqrt{2πy}} e^{−y/2} , y > 0

Then

E(X^{2}) = E(Y ) = ∫_{0}^{∞} yf_{Y}(y)dy = \frac{1}{\sqrt{2π}} ∫_{0}^{∞} \sqrt{y}e^{−y/2} dy = 1

beamer-tu-logo

On the other hand, we can use Theorem 2.1.5 to obtain the pdf of Y = −logX:

f_{Y}(y) = f_{X}(e^{−y} )|e^{−y} | = e^{−y} , y > 0

Then

E[g(X)] = E(Y ) = ∫_{−∞}^{∞} yf_{Y}(y)dy = ∫_{0}^{∞} ye^{−y} dy = 1

Example

Suppose that X has the standard normal pdf

f_{X}(x) = \frac{1}{\sqrt{2π}} e^{−x^{2}/2} , x ∈ R

Consider g(X) = X^{2}. By Example 2.1.9, Y = X^{2} has the chi-square pdf

f_{Y}(y) = \frac{1}{\sqrt{2πy}} e^{−y/2} , y > 0

Then

E(X^{2}) = E(Y ) = ∫_{0}^{∞} yf_{Y}(y)dy = \frac{1}{\sqrt{2π}} ∫_{0}^{∞} \sqrt{y}e^{−y/2} dy = 1

beamer-tu-logo

Theorem

Let F be the cdf of a random variable X. Then

E|X| = ∫_{0}^{∞} [1 − F(x)]dx + ∫_{−∞}^{0} F(x)dx

and E|X| < ∞ iff both integrals are finite. In the case where E|X| < ∞,

E(X) = ∫_{0}^{∞} [1 − F(x)]dx − ∫_{−∞}^{0} F(x)dx

Proof.

We only consider the case where X has a pdf or pmf, and give a proof when X has a pdf (the proof for the case where X has a pmf is similar). When X has a pdf f ,

∫_{0}^{∞} [1 − F(x)]dx = ∫_{0}^{∞} ∫_{x}^{∞} f (t)dtdx = ∫_{0}^{∞} ∫_{0}^{t}f (t)dxdt = ∫_{0}^{∞} tf (t)dt

∫_{0}^{∞} F(x)dx = ∫_{−∞}^{0} ∫_{−∞}^{x} f (t)dtdx = ∫_{−∞}^{0} ∫_{t}^{0}f (t)dxdt = ∫_{−∞}^{0} −tf (t)dt These calculations are valid regardless of whether the integrals are finite or not.

UW-Madison (Statistics)  Stat 609 Lecture 5  2014  9 / 11

beamer-tu-logo

By definition,

E|X| = ∫_{−∞}^{∞} |t|f (t)dt = ∫_{0}^{∞} tf (t)dt + ∫_{−∞}^{0} −tf (t)dt
= ∫_{0}^{∞} [1 − F(x)]dx + ∫_{−∞}^{0} F(x)dx

Since both integrals ≥ 0, E|X| < ∞ iff both integrals are finite. If E|X| < ∞, then both integrals are finite and

E(X) = ∫_{−∞}^{∞} tf (t)dt = ∫_{0}^{∞} tf (t)dt + ∫_{−∞}^{0} tf (t)dt
= ∫_{0}^{∞} [1 − F(x)]dx − ∫_{−∞}^{0} F(x)dx

Property

For any random variable X,

∑_{n=1}^{∞}P(|X| ≥ n) ≤ E|X| ≤ 1 + ∑_{n=1}^{∞}P(|X| ≥ n).

beamer-tu-logo

E|X| = ∫_{−∞}^{∞} |t|f (t)dt = ∫_{0}^{∞} tf (t)dt + ∫_{−∞}^{0} −tf (t)dt
= ∫_{0}^{∞} [1 − F(x)]dx + ∫_{−∞}^{0} F(x)dx

Since both integrals ≥ 0, E|X| < ∞ iff both integrals are finite. If E|X| < ∞, then both integrals are finite and

E(X) = ∫_{−∞}^{∞} tf (t)dt = ∫_{0}^{∞} tf (t)dt + ∫_{−∞}^{0} tf (t)dt
= ∫_{0}^{∞} [1 − F(x)]dx − ∫_{−∞}^{0} F(x)dx

Property

For any random variable X,

∑_{n=1}^{∞}P(|X| ≥ n) ≤ E|X| ≤ 1 + ∑_{n=1}^{∞}P(|X| ≥ n).

beamer-tu-logo

Using the previous theorem, we can prove this result regardless of whether X has a pdf (pmf) or not:

E|X| = ∫_{0}^{∞} [1 − F(x)]dx + ∫_{−∞}^{0} F(x)dx = ∫_{0}^{∞} [1 − F(x)]dx + ∫_{0}^{∞} F(−x)dx
= ∫_{0}^{∞} [1 − F(x) + F(−x)]dx = ∫_{0}^{∞} [P(X > x) + P(X ≤ −x)]dx
≤ ∫_{0}^{∞} [P(X ≥ x) + P(X ≤ −x)]dx = ∫_{0}^{∞} P(|X| ≥ x)dx
= ∑_{n=0}^{∞}∫_{n}^{n+1}P(|X| ≥ x)dx ≤ ∑_{n=0}^{∞}∫_{n}^{n+1}P(|X| ≥ n)dx
= ∑_{n=0}^{∞}P(|X| ≥ n) = 1 + ∑_{n=1}^{∞}P(|X| ≥ n)

Similarly,

E|X| ≥ ∫_{0}^{∞} [P(X > x) + P(X < −x)]dx = ∫_{0}^{∞} P(|X| > x)dx
= ∑_{n=0}^{∞}∫_{n}^{n+1}P(|X| > x)dx ≥ ∑_{n=0}^{∞}P(|X| ≥ n + 1) = ∑_{n=1}^{∞}P(|X| ≥ n)