beamer-tu-logo

Lecture 24: Hierarchical models

It is often easier to model a practical situation by thinking of things in a hierarchy.

Example 4.4.1 (binomial-Poisson hierarchy)

An insect lays many eggs, each surviving with probability p. On the average, how many eggs will survive? Let Y be the number of eggs and X be the number of survivors; both are random variables. We can model this situation by first modeling the distribution of Y ; given Y , we then model the distribution of X|Y . We can then obtain the joint distribution of (X,Y ) and marginal distributions of X and Y . We can model the number of eggs by a Poisson distribution, i.e., Y ∼ Poisson(λ), where λ > 0 is the average of eggs. Given Y , we can model the number of survivors as X|Y ∼ binomial(p,Y ).

UW-Madison (Statistics)  Stat 609 Lecture 24  2014  1 / 11

beamer-tu-logo

What is the marginal distribution of X? For x = 0,1,2,...,

P(X = x) = ∑_{y=x}^{∞}P(X = x,Y = y)  x ≤ y
= ∑_{y=x}^{∞}P(X = x|Y = y)P(Y = y)

= ∑_{y=x}^{∞} \binom{y}{x} p^{x}(1 − p)^{y−x} \frac{e^{−λ} λ^{y}}{y!}

= \frac{(λp)^{x}e^{−λ}}{x!} ∑_{y=x}^{∞} \frac{[(1 − p)λ]^{y−x}}{(y − x)!}

= \frac{(λp)^{x}e^{−λ}}{x!} ∑_{t=0}^{∞} \frac{[(1 − p)λ]^{t}}{t!}  t = y − x

= \frac{(λp)^{x}e^{−λ}}{x!} e^{(1−p)λ} = \frac{e^{−pλ} (λp)^{x}}{x!}

The distribution of X is Poisson(pλ)!

UW-Madison (Statistics)  Stat 609 Lecture 24  2014  2 / 11

beamer-tu-logo

Then, E(X) = pλ, which can also be obtained using

E(X) = E[E(X|Y )] = E(pY ) = pE(Y ) = pλ

without using the marginal distribution of X. If we begin with our model by saying that X ∼ Poisson(θ), then θ = pλ with Y playing no role at all. Introducing Y in the hierarchy was mainly aid our understanding of the model.

Example (binomial-binomial hierarchy)

A very similar hierarchical model can be described as follows. A market survey is conducted to study whether a new product is preferred over the product currently available in the market (old product). The survey is conducted by mail. Questionnaires are sent along with the sample products (both new and old) to N customers randomly selected from a population. Each customer is asked to fill out the questionnaire and return it, with response 1 (new is better than old) or 0 (otherwise).

UW-Madison (Statistics)  Stat 609 Lecture 24  2014  3 / 11

beamer-tu-logo

Then, E(X) = pλ, which can also be obtained using

E(X) = E[E(X|Y )] = E(pY ) = pE(Y ) = pλ

without using the marginal distribution of X. If we begin with our model by saying that X ∼ Poisson(θ), then θ = pλ with Y playing no role at all. Introducing Y in the hierarchy was mainly aid our understanding of the model.

Example (binomial-binomial hierarchy)

A very similar hierarchical model can be described as follows. A market survey is conducted to study whether a new product is preferred over the product currently available in the market (old product). The survey is conducted by mail. Questionnaires are sent along with the sample products (both new and old) to N customers randomly selected from a population. Each customer is asked to fill out the questionnaire and return it, with response 1 (new is better than old) or 0 (otherwise).

UW-Madison (Statistics)  Stat 609 Lecture 24  2014  3 / 11

beamer-tu-logo

Let X be the number of ones in the returned questionnaires. What is the distribution of X? If every customer returns the questionnaire, then (from elementary probability) X ∼ binomial(p,N) (assuming that the population is large enough so that customers respond independently), where p ∈ (0,1) is the overall rate of customers who prefer the new product. Some customers, however, do not return the questionnaires. Let Y be the number of customers who respond. If customers respond independently with the same probability π ∈ (0,1), then Y ∼ binomial(π,N). Given Y = y (an integer between 0 and N), X|Y = y ∼ (p,y) if y ≥ 1 and the point mass at 0 if y = 0. For x = 0,1,...,N,

P(X = x) = ∑_{k=x}^{N}P(X = x,Y = k) = ∑_{k=x}^{N}P(X = x|Y = k)P(Y = k)

beamer-tu-logo

= ∑_{k=x}^{N} \binom{k}{x} p^{x}(1 − p)^{k−x} \binom{N}{k} π^{k}(1 − π)^{N−k}

= \binom{N}{x} (πp)^{x}(1 − πp)^{N−x} ∑_{k=x}^{N}\binom{N − x}{k − x} ( \frac{π − πp}{1 − πp} ) ^{k−x}( \frac{1 − π}{1 − πp} ) ^{N−k}
= \binom{N}{x} (πp)^{x}(1 − πp)^{N−x}

It turns out that the marginal distribution of X is the binomial(πp,N) distribution.

Hierarchical models can have more than two stages. The advantage is that complicated processes may be modeled by a sequence of relatively simple models placed in a hierarchy. Conditional distributions play a central role. The random variables in hierarchical models may be all discrete (as in our previous examples), all continuous, or some discrete and some continuous.

UW-Madison (Statistics)  Stat 609 Lecture 24  2014  5 / 11

beamer-tu-logo

= \binom{N}{x} (πp)^{x}(1 − πp)^{N−x} ∑_{k=x}^{N}\binom{N − x}{k − x} ( \frac{π − πp}{1 − πp} ) ^{k−x}( \frac{1 − π}{1 − πp} ) ^{N−k}
= \binom{N}{x} (πp)^{x}(1 − πp)^{N−x}

It turns out that the marginal distribution of X is the binomial(πp,N) distribution.

Hierarchical models can have more than two stages. The advantage is that complicated processes may be modeled by a sequence of relatively simple models placed in a hierarchy. Conditional distributions play a central role. The random variables in hierarchical models may be all discrete (as in our previous examples), all continuous, or some discrete and some continuous.

UW-Madison (Statistics)  Stat 609 Lecture 24  2014  5 / 11

beamer-tu-logo

More general joint pdf’s and conditional distributions

So far we have considered only the situation where all random variables are continuous or all are discrete. What if X is a continuous random variable and Y is a discrete random variable on Y ? We can define the joint “pdf” to be a function f (x,y) satisfying

P(X ≤ x,Y = y) = ∫_{−∞}^{x} f (t,y)dt, x ∈ R, y ∈ Y

Then the marginal pdf of X and pmf of Y are respectively

f_{X}(x) = ∑_{y∈Y} f (x,y) and f_{Y}(y) = ∫_{−∞}^{∞} f (x,y)dx

The conditional pdf of X|Y or pmf of Y |X can be defined as before. Similarly we can deal with the situation where X is a continuous random vector and Y is a discrete random vector. If X or Y is neither discrete nor continuous, we can still define conditional distributions. But it involves higher level mathematics.

beamer-tu-logo

Example.

Suppose that X is a continuous random variable and Y is a discrete random variable with Y = {0,1}, and the joint pdf is

f (x,y) = \{ \begin{matrix}\frac{α}{\sqrt{2π}} e^{−x^{2}/2} & y = 0, \\ \frac{1−α}{2} e^{−|x|} & y = 1, \end{matrix} x ∈ R

where 0 < α < 1. Then f_{X}(x) = f (x,0) + f (x,1) = \frac{α}{\sqrt{2π}} e^{−x^{2}/2} + \frac{1 − α}{2} e^{−|x|} x ∈ R

f_{Y}(y) = ∫_{−∞}^{∞} f (x,y)dx = \{ \begin{matrix}α & y = 0 \\ 1 − α & y = 1 \end{matrix}

f (y|x) = \frac{f (x,y)}{f_{X}(x)} = \{ \begin{matrix}\frac{α}{\sqrt{2π}} e^{−x^{2}/2} / ( \frac{α}{\sqrt{2π}} e^{−x^{2}/2} + \frac{1−α}{2} e^{−|x|} ) & y = 0 \\ \frac{1−α}{2} e^{−|x|} / ( \frac{α}{\sqrt{2π}} e^{−x^{2}/2} + \frac{1−α}{2} e^{−|x|} ) & y = 1 \end{matrix}

beamer-tu-logo

f (x|y) = \frac{f (x,y)}{f_{Y}(y)} = \{ \begin{matrix}\frac{1}{\sqrt{2π}} e^{−x^{2}/2} & y = 0 \\ \frac{1}{2}e^{−|x|} & y = 1 \end{matrix} x ∈ R

Mixture distributions

In the previous example, the pdf f_{X} is referred to as a mixture distribution (or pdf), since it is a convex combination of two distributions (pdf’s).

Example 4.4.5 (three-stage hierarchy)

Consider a generalization of Example 4.4.1, where instead of one mother insect there are a large number of mothers and one mother is chosen at random from a (possibly continuous) distribution G.

The following three-stage hierarchy may be more appropriate:

number of survivors X from Y eggs  X|Y ∼ binomial(p,Y )
number of eggs Y from a given mother Z  Y |Z ∼ Poisson(Z)
a particular mother Z Z ∼ G
UW-Madison (Statistics)  Stat 609 Lecture 24  2014  8 / 11

beamer-tu-logo

f (x|y) = \frac{f (x,y)}{f_{Y}(y)} = \{ \begin{matrix}\frac{1}{\sqrt{2π}} e^{−x^{2}/2} & y = 0 \\ \frac{1}{2}e^{−|x|} & y = 1 \end{matrix} x ∈ R

Mixture distributions

In the previous example, the pdf f_{X} is referred to as a mixture distribution (or pdf), since it is a convex combination of two distributions (pdf’s).

Example 4.4.5 (three-stage hierarchy)

Consider a generalization of Example 4.4.1, where instead of one mother insect there are a large number of mothers and one mother is chosen at random from a (possibly continuous) distribution G.

The following three-stage hierarchy may be more appropriate:

number of survivors X from Y eggs  X|Y ∼ binomial(p,Y )
number of eggs Y from a given mother Z  Y |Z ∼ Poisson(Z)
a particular mother Z Z ∼ G
UW-Madison (Statistics)  Stat 609 Lecture 24  2014  8 / 11

beamer-tu-logo

f (x|y) = \frac{f (x,y)}{f_{Y}(y)} = \{ \begin{matrix}\frac{1}{\sqrt{2π}} e^{−x^{2}/2} & y = 0 \\ \frac{1}{2}e^{−|x|} & y = 1 \end{matrix} x ∈ R

Mixture distributions

In the previous example, the pdf f_{X} is referred to as a mixture distribution (or pdf), since it is a convex combination of two distributions (pdf’s).

Example 4.4.5 (three-stage hierarchy)

Consider a generalization of Example 4.4.1, where instead of one mother insect there are a large number of mothers and one mother is chosen at random from a (possibly continuous) distribution G.

The following three-stage hierarchy may be more appropriate:

number of survivors X from Y eggs  X|Y ∼ binomial(p,Y )
number of eggs Y from a given mother Z  Y |Z ∼ Poisson(Z)
a particular mother Z Z ∼ G
UW-Madison (Statistics)  Stat 609 Lecture 24  2014  8 / 11

beamer-tu-logo

Suppose that Z ∼ exponential(β). Then

E(X) = E[E(X|Y )] = pE(Y ) = pE[E(Y |Z)] = pE(Z) = pβ The three-stage model can be thought of as a two-stage model by combining the last two stages; we just need to obtain the marginal distribution of Y .

P(Y = y) = ∫_{0}^{∞} f (y|z)f_{Z}(z)dz = ∫_{0}^{∞} \frac{e^{−z} z^{y}}{y!} \frac{1}{β} e^{−z/β} dz
= \frac{1}{βy!} ∫_{0}^{∞} z^{y}e^{−z(1+β^{−1} )} dz = \frac{1}{βy!}Γ(y + 1)\frac{1}{(1 + β^{−1} )^{y+1}}
= \frac{1}{β} \frac{β^{y+1}}{(1 + β)^{y+1}} = \frac{1}{1 + β} ( 1 − \frac{1}{1 + β} ) ^{y}

which is the geometric distribution with probability (1 + β)^{−1} , i.e.,

number of survivors X from Y eggs  X|Y ∼ binomial(p,Y )
number of eggs Y  Y ∼ geometric((1 + β)^{−1} )

However, the three-stage model is easier to understand.

UW-Madison (Statistics)  Stat 609 Lecture 24  2014  9 / 11

beamer-tu-logo

Calculation of mean and variance

Aside from the advantage in understanding models, hierarchical models can often make the calculation of mean and variance easier. Let X be a random variable having a pdf that is the noncentral chisquare pdf with noncentrality parameter λ and degrees of freedom n:

f_{X}(x) = ∑_{k=1}^{∞} \frac{x^{n/2+k−1} e^{−x/2}}{Γ(n/2 + k)2^{n/2+k}} \frac{(λ/2)^{k}e^{−λ/2}}{k!} x > 0

Calculating E(X) is directly using this pdf not easy. However, the pdf is the marginal pdf of X in the following two-stage model:

X|K ∼ chi square with degrees of freedom n + 2K
K ∼ Poisson(λ/2)

Then

E(X) = E[E(X|K )] = E(n + 2K ) = n + 2E(K ) = n + λ
Var(X) = E[Var(X|K )] + Var(E(X|K )) = E(2n + 4K ) + Var(n + 2K )
= 2n + 4E(K ) + 4Var(K ) = 2n + 4λ
UW-Madison (Statistics)  Stat 609 Lecture 24  2014  10 / 11

Example 4.4.6.

Suppose that X|P ∼ binomial(n,P), P ∼ beta(α,β)

How to calculate E(X) and Var(X)?

E(X) = E[E(X|P)] = E(nP) = nE(P) = \frac{nα}{α + β}

Using the variance formula derived previously,

Var(X) = Var(E(X|P)) + E[Var(X|P)]
= Var(nP) + E[nP(1 − P)]
= n^{2}Var(P) + nE(P) − nE(P^{2})
= \frac{n^{2}αβ}{(α + β)^{2}(α + β + 1)} + \frac{nα}{α + β} − \frac{nα(α + 1)}{(α + β)(α + β + 1)}
= \frac{nαβ(α + β + n)}{(α + β)^{2}(α + β + 1)}

beamer-tu-logo UW-Madison (Statistics)  Stat 609 Lecture 24  2014  11 / 11