beamer-tu-logo

Lecture 13: Inequalities

Inequalities are useful for statistical theory.

Theorem 3.6.1 (Chebychev’s inequality)

Let X be a random variable and let g(x) be a nonnegative function. For any r > 0,

P(g(X) ≥ r) ≤ \frac{E[g(X)]}{r}

Proof.

Assuming that X has a pdf (if X has a pmf, we replace integral by summation), we have

P(g(X) ≥ r) = ∫_{−∞}^{∞} g(x)f_{X}(x)dx ≥ ∫_{−∞}^{∞} I({g(x) ≥ r})g(x)f_{X}(x)dx
≥ r ∫_{−∞}^{∞} I({g(x) ≥ r})f_{X}(x)dx = r ∫_{{g(x)≥r}} f_{X}(x)dx
= rP(g(X) ≥ r)

beamer-tu-logo

Lecture 13: Inequalities

Inequalities are useful for statistical theory.

Theorem 3.6.1 (Chebychev’s inequality)

Let X be a random variable and let g(x) be a nonnegative function. For any r > 0,

P(g(X) ≥ r) ≤ \frac{E[g(X)]}{r}

Proof.

Assuming that X has a pdf (if X has a pmf, we replace integral by summation), we have

P(g(X) ≥ r) = ∫_{−∞}^{∞} g(x)f_{X}(x)dx ≥ ∫_{−∞}^{∞} I({g(x) ≥ r})g(x)f_{X}(x)dx
≥ r ∫_{−∞}^{∞} I({g(x) ≥ r})f_{X}(x)dx = r ∫_{{g(x)≥r}} f_{X}(x)dx
= rP(g(X) ≥ r)

beamer-tu-logo

Different forms of Chebychev’s inequality

If g is nondecreasing, then another form of Chebychev’s inequality is, for ϵ > 0,

P(|X| ≥ ϵ) ≤ \frac{E[g(X)]}{g(ϵ)}

Suppose that X has expectation μ and variance σ^{2}. For g(x) = (x − μ)^{2}/σ^{2}, we have

P(|X − μ| ≥ tσ) = P ( \frac{(X − μ)^{2}}{σ^{2}} ≥ t^{2}) ≤ \frac{1}{t^{2}}E \frac{(X − μ)^{2}}{σ^{2}} = \frac{1}{t^{2}}

If X has a finite kth moment with an integer k, then, for t > 0,

P(|X − μ| ≥ t) ≤ \frac{E|X − μ|^{k}}{t^{k}}

If X has a finite mgf M_{X}(t) for t ∈ (−h,h), then, for r > 0 and t > 0,

P(X ≥ r) ≤ \frac{E(e^{tX})}{e^{tr}} = \frac{M_{X}(t)}{e^{tr}}, P(X ≤ −r) ≤ \frac{E(e^{−tX})}{e^{tr}} = \frac{M_{X}(−t)}{e^{tr}}

P(|X| ≥ r) ≤ \frac{M_{X}(t) + M_{X}(−t)}{e^{tr}}

UW-Madison (Statistics)  Stat 609 Lecture 13  2014  2 / 11

beamer-tu-logo

Chebychev’s inequality is useful, but sometimes it is too loose because it does not require much from the distribution of X except some moment conditions. More useful probability inequality can be derived when we know something about the distribution of X.

Example 3.6.3 (A normal probability inequality

If Z ∼ N(0,1), then

P(|Z| ≥ t) ≤ \frac{\sqrt{2}e^{−t^{2}/2}}{\sqrt{π}t} , t > 0

because

P(|Z| ≥ t) = 2P(Z ≥ t) = \frac{2}{\sqrt{2π}} ∫_{t}^{∞} e^{−z^{2}/2} dz
≤ \begin{matrix}√ & \overline{2} \\ √ & \overline{π} \end{matrix} ∫_{t}^{∞} \frac{z}{t}e^{−z^{2}/2} dz = \frac{\sqrt{2}}{\sqrt{π}t}e^{−z^{2}/2} |_{∞}^{t}
= \frac{\sqrt{2}e^{−t^{2}/2}}{\sqrt{π}t}
UW-Madison (Statistics)  Stat 609 Lecture 13  2014  3 / 11

beamer-tu-logo

Comparing this inequality with Chebychev’s inequality obtained previously,

P(|Z| ≥ t) ≤ \frac{M_{X}(t) + M_{X}(−t)}{e^{t^{2}}} = \frac{2e^{t^{2}/2}}{e^{t^{2}}} = 2e^{−t^{2}/2}

we find that this inequality is more accurate by a factor of \sqrt{2π}t, which is large when t is large.

Cauchy-Schwartz’s inequality

This is another simple but very useful inequality. If X and Y are random variables with a^{2} = E(X^{2}) < ∞ and b^{2} = E(Y^{2}) < ∞, then the following Cauchy-Schwartz’s inequality holds:

[E(XY )]^{2} ≤ E(X^{2})E(Y^{2})

with equality holds iff P(X = cY ) = 1 for a constant c. In fact, we also have

[E|XY |]^{2} ≤ E(X^{2})E(Y^{2})

beamer-tu-logo

Comparing this inequality with Chebychev’s inequality obtained previously,

P(|Z| ≥ t) ≤ \frac{M_{X}(t) + M_{X}(−t)}{e^{t^{2}}} = \frac{2e^{t^{2}/2}}{e^{t^{2}}} = 2e^{−t^{2}/2}

we find that this inequality is more accurate by a factor of \sqrt{2π}t, which is large when t is large.

Cauchy-Schwartz’s inequality

This is another simple but very useful inequality. If X and Y are random variables with a^{2} = E(X^{2}) < ∞ and b^{2} = E(Y^{2}) < ∞, then the following Cauchy-Schwartz’s inequality holds:

[E(XY )]^{2} ≤ E(X^{2})E(Y^{2})

with equality holds iff P(X = cY ) = 1 for a constant c. In fact, we also have

[E|XY |]^{2} ≤ E(X^{2})E(Y^{2})

beamer-tu-logo

Proof of Cauchy-Schwartz’s inequality

For any t > 0 and s > 0,

2\sqrt{st} ≤ s + t because (\sqrt{s} − \sqrt{t})^{2} ≥ 0

Letting s = X^{2}/a^{2} and t = Y^{2}/b^{2}, we obtain

\frac{|XY |}{ab}≤ \frac{X^{2}}{2a^{2}} + \frac{Y^{2}}{2b^{2}}  hence  \frac{E|XY |}{ab}≤ \frac{E(X^{2})}{2a^{2}}+ \frac{E(Y^{2})}{2b^{2}} = 1

which means

[E|XY |]^{2} ≤ a^{2}b^{2} = E(X^{2})E(Y^{2})

The other inequality follows since

[E(XY )]^{2} ≤ [E|XY |]^{2} ≤ E(X^{2})E(Y^{2})

We next consider what happens if the equality holds. If [E|XY |]^{2} = E(X^{2})E(Y^{2}), then

\frac{E|XY |}{ab}= \frac{E(X^{2})}{2a^{2}}+ \frac{E(Y^{2})}{2b^{2}} i.e., E ( \frac{|X|}{a}− \frac{|Y |}{b}) ^{2} = 0

beamer-tu-logo

It follows from a result established previously that

P ( \frac{|X|}{a}− \frac{|Y |}{b} = 0) = 1 i.e., P ( |X| = \frac{a}{b}|Y |) = 1

Finally, consider the situation where [E(XY )]^{2} = E(X^{2})E(Y^{2}). Since [E(XY )]^{2} ≤ [E|XY |]^{2}, this implies

[E(XY )]^{2} = [E|XY |]^{2} = E(X^{2})E(Y^{2})

and, by the early proof, P(|X| = \frac{a}{b}|Y |) = 1. From [E(XY )]^{2} = [E|XY |]^{2}, we must have ±E(XY ) = E|XY |. Suppose E(XY ) = E|XY | (the proof for −E(XY ) = E|XY | is similar). Since |XY | − XY ≥ 0, E|XY | − E(XY ) = 0 implies P(|XY | = XY ) = 1. Combining this with the early result, we must have P(X = \frac{a}{b}Y ) = 1.

Cauchy-Schwartz’s inequality is a special case of the following result. Hölder’s inequality: If p and q are positive constants satisfying p > 1 and p^{−1} + q^{−1} = 1 and X and Y are random variables, then

E|XY | ≤ (E|X|^{p})^{1/p}(E|Y |^{q})^{1/q}

beamer-tu-logo

P ( \frac{|X|}{a}− \frac{|Y |}{b} = 0) = 1 i.e., P ( |X| = \frac{a}{b}|Y |) = 1

Finally, consider the situation where [E(XY )]^{2} = E(X^{2})E(Y^{2}). Since [E(XY )]^{2} ≤ [E|XY |]^{2}, this implies

[E(XY )]^{2} = [E|XY |]^{2} = E(X^{2})E(Y^{2})

and, by the early proof, P(|X| = \frac{a}{b}|Y |) = 1. From [E(XY )]^{2} = [E|XY |]^{2}, we must have ±E(XY ) = E|XY |. Suppose E(XY ) = E|XY | (the proof for −E(XY ) = E|XY | is similar). Since |XY | − XY ≥ 0, E|XY | − E(XY ) = 0 implies P(|XY | = XY ) = 1. Combining this with the early result, we must have P(X = \frac{a}{b}Y ) = 1.

Cauchy-Schwartz’s inequality is a special case of the following result. Hölder’s inequality: If p and q are positive constants satisfying p > 1 and p^{−1} + q^{−1} = 1 and X and Y are random variables, then

E|XY | ≤ (E|X|^{p})^{1/p}(E|Y |^{q})^{1/q}

beamer-tu-logo

Using Hölder’s inequality, we can obtain the following two inequalities. Liapounov’s inequality: If r and s are constants satisfying 1 ≤ r ≤ s and X is a random variable, then

(E|X|^{r})^{1/r} ≤ (E|X|^{s})^{1/s}

Minkowski’s inequality: If p ≥ 1 is a constant and X and Y are random variables, then

(E|X + Y |^{p})^{1/p} ≤ (E|X|^{p})^{1/p} + (E|Y |^{p})^{1/p}

Convex set and function

A set A ⊂ R^{k} is convex iff x ∈ A, y ∈ A, and t ∈ (0,1) imply

tx + (1 − t)y ∈ A

A function g from s convex A ⊂ R^{k} to R is convex iff x ∈ A, y ∈ A, and t ∈ (0,1) imply

g(tx + (1 − t)y) ≤ t g(x) + (1 − t)g(y)

and g is strictly convex iff the previous inequality holds with ≤ replaced by <.

UW-Madison (Statistics)  Stat 609 Lecture 13  2014  7 / 11

beamer-tu-logo

If g is twice differentiable on a convex A, then a necessary and sufficient condition for g to be convex (or strictly convex) is that the k × k second order partial derivative matrix ∂^{2}g/∂x∂x′is nonnegative definite (or positive definite). If g is convex, then −g is concave. The following is a very useful inequality in statistics.

beamer-tu-logo

Jensen’s inequality

If g is a convex function on a convex A ⊂ R and X is a random variable with P(X ∈ A) = 1, then

g(E(X)) ≤ E[g(X)]

provided that the expectations exist. If g is strictly convex, then ≤ in the previous inequality can be replaced by < unless P(g(X) = c) = 1 for a constant c.

Jensen’s inequality also holds for a convex g defined on A ⊂ R^{k} and a random vector X defined on R^{k} introduced in Chapter 4.

Proof of Jensen’s inequality

Let l(x) = a+bx be the tangent line to g(x) at g(E(X)) (see the figure). Since g is convex, g(x) ≥ a + bx for all x and, hence,

E[g(X)] ≥ E(a + bX) = a + bE(X) = l(E(X)) = g(E(X)).

If E[g(X)] = g(E(X)), then P(g(X) = a + bX) = 1, which cannot occur if g is strictly convex and g(X) is not a constant.

UW-Madison (Statistics)  Stat 609 Lecture 13  2014  9 / 11

beamer-tu-logo

Jensen’s inequality

If g is a convex function on a convex A ⊂ R and X is a random variable with P(X ∈ A) = 1, then

g(E(X)) ≤ E[g(X)]

provided that the expectations exist. If g is strictly convex, then ≤ in the previous inequality can be replaced by < unless P(g(X) = c) = 1 for a constant c.

Jensen’s inequality also holds for a convex g defined on A ⊂ R^{k} and a random vector X defined on R^{k} introduced in Chapter 4.

Proof of Jensen’s inequality

Let l(x) = a+bx be the tangent line to g(x) at g(E(X)) (see the figure). Since g is convex, g(x) ≥ a + bx for all x and, hence,

E[g(X)] ≥ E(a + bX) = a + bE(X) = l(E(X)) = g(E(X)).

If E[g(X)] = g(E(X)), then P(g(X) = a + bX) = 1, which cannot occur if g is strictly convex and g(X) is not a constant.

UW-Madison (Statistics)  Stat 609 Lecture 13  2014  9 / 11

beamer-tu-logo

Jensen’s inequality

If g is a convex function on a convex A ⊂ R and X is a random variable with P(X ∈ A) = 1, then

g(E(X)) ≤ E[g(X)]

provided that the expectations exist. If g is strictly convex, then ≤ in the previous inequality can be replaced by < unless P(g(X) = c) = 1 for a constant c.

Jensen’s inequality also holds for a convex g defined on A ⊂ R^{k} and a random vector X defined on R^{k} introduced in Chapter 4.

Proof of Jensen’s inequality

Let l(x) = a+bx be the tangent line to g(x) at g(E(X)) (see the figure). Since g is convex, g(x) ≥ a + bx for all x and, hence,

E[g(X)] ≥ E(a + bX) = a + bE(X) = l(E(X)) = g(E(X)).

If E[g(X)] = g(E(X)), then P(g(X) = a + bX) = 1, which cannot occur if g is strictly convex and g(X) is not a constant.

UW-Madison (Statistics)  Stat 609 Lecture 13  2014  9 / 11

beamer-tu-logo

Examples

The function g(x) = x^{−1} is strictly convex. Hence,

(EX)^{−1} < E(X^{−1} )

unless P(X = c) = 1 for a constant c. The function g(x) = −logx is strictly convex (logx is strictly concave). Then

−log(EX) < −E(logX) i.e., E(logX) < log(EX) unless P(X = c) = 1 for a constant c. Let f and g be positive functions satisfying 0 < ∫_{−∞}^{∞} g(x)dx ≤ ∫_{−∞}^{∞} f (x)dx = 1. We want to show that

∫_{−∞}^{∞} f (x)log \frac{g(x)}{f (x)}dx ≤ 0

Note that f is a pdf. Let X ∼ f and Y = \frac{g(X)}{f (X)}.

beamer-tu-logo

By Jensen’s inequality,

∫_{−∞}^{∞} f (x)log \frac{g(x)}{f (x)}dx = E ( log \frac{g(X)}{f (X)}) = E(logY )
≤ log(EY ) = log( E \frac{g(X)}{f (X)})
= log( ∫_{−∞}^{∞} \frac{g(x)}{f (x)}f (x)dx) = log( ∫_{−∞}^{∞} g(x)dx)
≤ log( ∫_{−∞}^{∞} f (x)dx) = log(1)
= 0

where the last inequality follows from the fact that logx is increasing. Also, the strict < holds unless P(f (X) = g(X)) = 1.