beamer-tu-logo

Lecture 18: Sum of random variables

In applications, we frequently consider a sum of random variables or, in many cases, a sum of independent random variables.

The pdf of the sum of two random variables (convolution)

Let X and Y be random variables having joint pdf f (x,y). For any t ∈ R,

P(X + Y ≤ t) = ∫ ∫_{x+y≤t} f (x,y)dxdy = ∫_{−∞}^{∞} [∫_{−∞}^{t−x} f (x,y)dy]dx

Hence, the pdf of the sum X + Y is

f_{X+Y}(t) = \frac{d}{dt}P(X +Y ≤ t) =∫_{−∞}^{∞} \frac{d}{dt}[∫_{−∞}^{t−x} f (x,y)dy]dx =∫_{−∞}^{∞} f (x,t −x)dx

Similarly,

f_{X+Y}(t) = ∫_{−∞}^{∞} f (t − y,y)dy

The pdf f_{X+Y} is called a convolution. The interchange of differentiation and integration can be justified.

UW-Madison (Statistics)  Stat 609 Lecture 18  2014  1 / 12

beamer-tu-logo

If X and Y are independent and f_{X} and f_{Y} are their marginal pdf’s, then

f_{X+Y}(t) = ∫_{−∞}^{∞} f_{X}(x)f_{Y}(t − x)dx = ∫_{−∞}^{∞} f_{X}(t − y)f_{Y}(y)dy

Example

If X ∼ uniform(0,1) and Y ∼ exponential(0,1) are independent, what is the pdf of X + Y ?

f_{X+Y}(t) = ∫_{−∞}^{∞} f_{X}(x)f_{Y}(t − x)dx = ∫_{0<x<1,x<t} e^{−(t−x)} dx

= \{ \begin{matrix}∫_{0}^{1}e^{−(t−x)} dx = (e − 1)e^{−t} & t > 1 \\ ∫_{0}^{t}e^{−(t−x)} dx = 1 − e^{−t} & 0 < t ≤ 1 \\ 0 & t ≤ 0 \end{matrix}

When f_{X} or f_{Y} is 0 in some regions, we need to be careful about the integration limits. Sometimes, one of the two formulas is easier to work with.

f_{X+Y}(t) = ∫_{−∞}^{∞} f_{X}(t − y)f_{Y}(y)dy = ∫_{0<t−y<1,y>0} e^{−y} dy =?

Example

Suppose that (X,Y ) has the following bivariate normal pdf:

f (x,y) = \frac{exp( −\frac{x^{2}}{2(1−ρ^{2})} + \frac{ρxy}{(1−ρ^{2})} − \frac{y^{2}}{2(1−ρ^{2})} )}{2π\sqrt{1 − ρ^{2}}} (x,y) ∈ R^{2}

What is the pdf of X + Y ?

f_{X+Y}(t) = \frac{1}{2π\sqrt{1 − ρ^{2}}} ∫_{−∞}^{∞} exp( −\frac{x^{2}}{2(1 − ρ^{2})} + \frac{ρx(t − x)}{(1 − ρ^{2})} − \frac{(t − x)^{2}}{2(1 − ρ^{2})} ) dx
= \frac{1}{2π\sqrt{1 − ρ^{2}}} ∫_{−∞}^{∞} exp( −\frac{x^{2} + 2ρx^{2} + x^{2} − 2ρtx − 2tx + t^{2}}{2(1 − ρ^{2})} ) dx
= \frac{1}{2π\sqrt{1 − ρ^{2}}} ∫_{−∞}^{∞} exp( −\frac{(x − t/2)^{2}}{1 − ρ} − \frac{t^{2}}{4(1 + ρ)} ) dx
= \frac{1}{2\sqrt{π(1 + ρ)}} exp( −\frac{t^{2}}{4(1 + ρ)} )

which is N(0,2(1 + ρ)).

beamer-tu-logo UW-Madison (Statistics)  Stat 609 Lecture 18  2014  3 / 12

beamer-tu-logo

The pmf of the sum of two discrete random variables

We now turn to discrete random variables X and Y with joint pmf f (x,y). By definition, the pmf of X + Y is

f_{X+Y}(t) = P(X + Y = t) = ∑_{x+y≤t} f (x,y) = ∑_{x} f (x,t − x) = ∑_{y} f (t − y,y)

and if X and Y are independent with marginal pmf’s f_{X} and f_{Y}, then

f_{X+Y}(t) = ∑_{x} f_{X}(x)f_{Y}(t − x) = ∑_{y} f_{X}(t − y)f_{Y}(y)

However, in some cases it is more convenient to work out the probability P(X + Y = t) directly.

Example 4.3.1.

Let X ∼ Poisson(θ) and Y ∼ Poisson(λ) be independent Poisson random variables so that the joint pmf of (X,Y ) is

f (x,y) = \frac{θ^{x}e^{−θ}}{x!} \frac{λ^{y}e^{−λ}}{y!} , x = 0,1,2,...,y = 0,1,2,...

beamer-tu-logo

What is the pmf of U = X + Y ? Using the formula, for u = 0,1,2,...

f_{U}(u) = ∑_{y} f (u − y,y) = ∑_{y=0}^{u} \frac{θ^{u−y} e^{−θ}}{(u − y)!} \frac{λ^{y}e^{−λ}}{y!}

= \frac{e^{−(θ+λ)}}{u!} ∑_{y=0}^{u} \binom{u}{y} λ^{y}θ^{u−y} = \frac{e^{−(θ+λ)} (θ + λ)^{u}}{u!}

This is the pmf of Poisson(θ + λ). What is the joint pmf of U = X + Y and Y .

f_{U,Y}(u,y) = P(U = u,Y = y) = P(X + Y = u,Y = y)
= P(X = u − y,Y = v) = \frac{θ^{u−y} e^{−θ}}{(u − y)!} \frac{λ^{y}e^{−λ}}{y!}

for u = y,y + 1,y + 2,..., and y = 0,1,2,... U and Y are not independent, because

P(U = 0,Y = 1) = 0 ≠ P(U = 0)P(Y = 1) = e^{−(θ+λ)} λe^{−λ}

UW-Madison (Statistics)  Stat 609 Lecture 18  2014  5 / 12

beamer-tu-logo

To deal with a sum of independent random variables, it is convenient to use mgf’s.

Theorem 4.2.12.

If X_{1},...,X_{n} are independent random variables with mgf’s M_{X_{1}}(t),...,M_{X_{n}}(t), then the mgf of the sum T = X_{1} + ··· + X_{n} is

M_{T}(t) = M_{X_{1}}(t)··· M_{X_{n}}(t)

and M_{T}(t) < ∞ iff M_{X_{i}}(t) < ∞, i = 1,...,n.

Proof.

By definition,

M_{T}(t)= E(e^{tT}) = E(e^{t(X_{1}+··· +X_{n})}) = E(e^{tX_{1}}··· e^{tX_{n}})
= E(e^{tX_{1}})··· E(e^{tX_{n}}) = M_{X_{1}}(t)··· M_{X_{n}}(t)

where the 4th equality follows from the independence of X_{1},...,X_{n} and Theorem 4.6.12.

This result and the uniqueness theorem (Theorem 2.3.11) lead to many useful results concerning the sum of independent random variables.

UW-Madison (Statistics)  Stat 609 Lecture 18  2014  6 / 12

beamer-tu-logo

To deal with a sum of independent random variables, it is convenient to use mgf’s.

Theorem 4.2.12.

If X_{1},...,X_{n} are independent random variables with mgf’s M_{X_{1}}(t),...,M_{X_{n}}(t), then the mgf of the sum T = X_{1} + ··· + X_{n} is

M_{T}(t) = M_{X_{1}}(t)··· M_{X_{n}}(t)

and M_{T}(t) < ∞ iff M_{X_{i}}(t) < ∞, i = 1,...,n.

Proof.

By definition,

M_{T}(t)= E(e^{tT}) = E(e^{t(X_{1}+··· +X_{n})}) = E(e^{tX_{1}}··· e^{tX_{n}})
= E(e^{tX_{1}})··· E(e^{tX_{n}}) = M_{X_{1}}(t)··· M_{X_{n}}(t)

where the 4th equality follows from the independence of X_{1},...,X_{n} and Theorem 4.6.12.

This result and the uniqueness theorem (Theorem 2.3.11) lead to many useful results concerning the sum of independent random variables.

UW-Madison (Statistics)  Stat 609 Lecture 18  2014  6 / 12

beamer-tu-logo

Theorem 4.2.14.

If X ∼ N(μ,σ^{2}) and Y ∼ N(γ,τ^{2}) are independent, then T = X + Y ∼ N(μ + γ,σ^{2} + τ^{2}).

This is easy to show, because the mfg’s of X and Y are M_{X}(t) = e^{μt+σ^{2}t^{2}/2} and M_{Y}(t) = e^{γt+τ^{2}t^{2}/2}, and thus the mgf of T is

M_{X}(t)M_{Y}(t) = e^{μt+σ^{2}t^{2}/2}e^{γt+τ^{2}t^{2}/2}= e^{(μ+γ)t+(σ^{2}+τ^{2})t^{2}/2}

which is the mgf of N(μ + γ,σ^{2} + τ^{2}). By the uniqueness theorem, we must have T ∼ N(μ + γ,σ^{2} + τ^{2}). Note that this result does not apply to the bivariate normal distribution example with ρ ≠ 0, since in that example X and Y are not independent.

The previous result can be generalized to more than two independent random variables having normal distributions: If X_{1},...,X_{k} are independent random variables, X_{i} ∼ N(μ_{i},σ_{i}^{2}), i = 1,...,k, then the sum X_{1} + ··· + X_{k} ∼ N(μ_{1} + ··· + μ_{k},σ_{1}^{2} + ··· + σ_{k}^{2}).

beamer-tu-logo

This type of results is called additivity of distributions; i.e., if X and Y have distributions in a class of distributions and X + Y has a distribution also in that class, then the distributions in this class are called additive distributions.

Additivity of the chi-square distributions

In Chapter 2 we showed that if Z ∼ N(0,1), then Z^{2} ∼ the chi-square distribution with degree of freedom 1. A more general result is as follows. If X_{i} ∼ N(μ_{i},σ_{i}^{2}), i = 1,...,k, are independent, then the distribution of

Y = ( \frac{X_{1} − μ_{1}}{σ_{1}} ) ^{2} + ··· + ( \frac{X_{k} − μ_{k}}{σ_{k}} ) ^{2}

is the chi-square distribution with degrees of freedom k. First, Z_{i} ∼ N(0,1), i = 1,...,k, and Z_{i}’s are independent. Second, each Z_{i}^{2} ∼ chi-square with degree of freedom 1. Third, the chi-square distribution with degrees of freedom r is the gamma(r/2,2) distribution. Finally, the result follows from the next result.

UW-Madison (Statistics)  Stat 609 Lecture 18  2014  8 / 12

beamer-tu-logo

Additivity of the gamma distributions

If X_{i} ∼ gamma(α_{i},β), i = 1,...,k, are independent, then the sum T = X_{1} + ··· + X_{k} ∼ gamma(α_{1} + ··· + α_{k},β). This is because the mgf of X_{i} is (1 − βt)^{−α_{i}} when t < β^{−1} . By Theorem 4.2.12, the mgf of T is

(1 − βt)^{−α_{1}} ··· (1 − βt)^{−α_{k}} = (1 − βt)^{−(α_{1}+··· +α_{k})} t < β^{−1} which is the mgf of gamma(α_{1} + ··· + α_{k},β).

A non-additive class of distributions

The class of all uniform distributions is not additive. If X ∼ uniform(0,1) and Y ∼ uniform(0,1) are independent and f (x) is the indicator function of the interval (0,1), then the pdf of X + Y is

f_{X+Y}(t) = ∫_{−∞}^{∞} f (t − y)f (y)dy = ∫_{0<t−y<1,0<y<1} dy

= \{ \begin{matrix}∫_{t−1}^{1} dy = 2 − t & 1 < t < 2 \\ ∫_{0}^{t}dy = t & 0 < t < 1 \\ 0 & otherwise \end{matrix}

beamer-tu-logo

Additivity of the gamma distributions

If X_{i} ∼ gamma(α_{i},β), i = 1,...,k, are independent, then the sum T = X_{1} + ··· + X_{k} ∼ gamma(α_{1} + ··· + α_{k},β). This is because the mgf of X_{i} is (1 − βt)^{−α_{i}} when t < β^{−1} . By Theorem 4.2.12, the mgf of T is

(1 − βt)^{−α_{1}} ··· (1 − βt)^{−α_{k}} = (1 − βt)^{−(α_{1}+··· +α_{k})} t < β^{−1} which is the mgf of gamma(α_{1} + ··· + α_{k},β).

A non-additive class of distributions

The class of all uniform distributions is not additive. If X ∼ uniform(0,1) and Y ∼ uniform(0,1) are independent and f (x) is the indicator function of the interval (0,1), then the pdf of X + Y is

f_{X+Y}(t) = ∫_{−∞}^{∞} f (t − y)f (y)dy = ∫_{0<t−y<1,0<y<1} dy

= \{ \begin{matrix}∫_{t−1}^{1} dy = 2 − t & 1 < t < 2 \\ ∫_{0}^{t}dy = t & 0 < t < 1 \\ 0 & otherwise \end{matrix}

beamer-tu-logo

The next result is useful if the mgf is not finite in a neighborhood of 0.

Theorem 4.2.12A.

If X_{1},...,X_{n} are independent random variables with chf’s ϕ_{X_{1}}(t),...,ϕ_{X_{n}}(t), then the chf of the sum T = X_{1} + ··· + X_{n} is

ϕ_{T}(t) = ϕ_{X_{1}}(t)··· ϕ_{X_{n}}(t) t ∈ R

Additivity of the Cauchy distribution

If U and V are independent random variables having the Cauchy(0,1) distribution, what is the distribution of U + V ? Since the mgf of a Cauchy distribution is not finite except at 0, we apply Theorem 4.2.12A. Note that the chf of Cauchy(0,1) is e^{−|t|}, t ∈ R. By Theorem 4.2.12A, the chf of U + V is e^{−|t|}e^{−|t|} = e^{−2|t|}, t ∈ R, which is the chf of Cauchy(0,2). By the uniqueness theorem, U + V ∼ Cauchy(0,2). Of course, the same result can be derived using the convolution, but the argument is much more complicated.

UW-Madison (Statistics)  Stat 609 Lecture 18  2014  10 / 12

beamer-tu-logo

The next result is useful if the mgf is not finite in a neighborhood of 0.

Theorem 4.2.12A.

If X_{1},...,X_{n} are independent random variables with chf’s ϕ_{X_{1}}(t),...,ϕ_{X_{n}}(t), then the chf of the sum T = X_{1} + ··· + X_{n} is

ϕ_{T}(t) = ϕ_{X_{1}}(t)··· ϕ_{X_{n}}(t) t ∈ R

Additivity of the Cauchy distribution

If U and V are independent random variables having the Cauchy(0,1) distribution, what is the distribution of U + V ? Since the mgf of a Cauchy distribution is not finite except at 0, we apply Theorem 4.2.12A. Note that the chf of Cauchy(0,1) is e^{−|t|}, t ∈ R. By Theorem 4.2.12A, the chf of U + V is e^{−|t|}e^{−|t|} = e^{−2|t|}, t ∈ R, which is the chf of Cauchy(0,2). By the uniqueness theorem, U + V ∼ Cauchy(0,2). Of course, the same result can be derived using the convolution, but the argument is much more complicated.

UW-Madison (Statistics)  Stat 609 Lecture 18  2014  10 / 12

As an exercise, we now use convolution to derive the same result. By the convolution formula, the pdf of U + V is

f_{U+V}(z) = ∫_{−∞}^{∞} \frac{1}{π(1 + w^{2})} \frac{1}{π[1 + (z − w)^{2}]} dw

After some calculation, we obtain that

\frac{1}{π^{2}(1 + w^{2})[1 + (z − w)^{2}]} = \frac{2}{π^{2}(z^{2} + 4)z} [ \frac{w}{1 + w^{2}} − \frac{w − z}{1 + (z − w)^{2}} ]

+\frac{1}{π^{2}(z^{2} + 4)} [ \frac{1}{1 + w^{2}} + \frac{1}{1 + (z − w)^{2}} ]

For every fixed z, when |w| is large enough,

| \frac{w}{1 + w^{2}} − \frac{w − z}{1 + (z − w)^{2}} | = | \frac{2(1 − w^{2} + zw)}{(1 + w^{2})[1 + (z − w)^{2}]} | ≤ \frac{4}{1 + (z − w)^{2}}

Hence the first term on the left hand side of the above expression is integrable and, by symmetry,

beamer-tu-logo UW-Madison (Statistics)  Stat 609 Lecture 18  2014  11 / 12

beamer-tu-logo

∫_{−∞}^{∞} [ \frac{w}{1 + w^{2}} − \frac{w − z}{1 + (z − w)^{2}} ] dw = 0

Then,

f_{U+V}(z) = \frac{1}{π^{2}(z^{2} + 4)} ∫_{−∞}^{∞} [ \frac{1}{1 + w^{2}} + \frac{1}{1 + (z − w)^{2}} ] dw
= \frac{1}{π^{2}(z^{2} + 4)} (π + π) = \frac{1}{2π[1 + (z/2)^{2}]}

This is the pdf of Cauchy(0,2) distribution.