beamer-tu-logo

Lecture 12: Exponential and location-scale families

Families of Distributions

In statistics we are interested in some families of distributions, i.e., some collections of distributions. For example, the family of binomial distributions with p ∈ (0,1) and a fixed n; the family of normal distributions with μ ∈ R and σ > 0.

Exponential families

A family of pdfs or pmfs indexed by θ is called an exponential family iff it can be expressed as

f_{θ}(x) = h(x)c(θ)exp( ∑_{i=1}^{k}w_{i}(θ)t_{i}(x)) , θ ∈ Θ,

where exp(x) = e^{x}, Θ is the set of all values of θ (parameter space), h(x) ≥ 0 and t_{1}(x),...,t_{k}(x) are functions of x (not depending on θ), and c(θ) > 0 and w_{1}(θ),...,w_{k}(θ) are functions of the possibly vector-valued θ (not depending on x). Note that the expression for f may not be unique.

UW-Madison (Statistics)  Stat 609 Lecture 12  2014  1 / 12

beamer-tu-logo

Lecture 12: Exponential and location-scale families

Families of Distributions

In statistics we are interested in some families of distributions, i.e., some collections of distributions. For example, the family of binomial distributions with p ∈ (0,1) and a fixed n; the family of normal distributions with μ ∈ R and σ > 0.

Exponential families

A family of pdfs or pmfs indexed by θ is called an exponential family iff it can be expressed as

f_{θ}(x) = h(x)c(θ)exp( ∑_{i=1}^{k}w_{i}(θ)t_{i}(x)) , θ ∈ Θ,

where exp(x) = e^{x}, Θ is the set of all values of θ (parameter space), h(x) ≥ 0 and t_{1}(x),...,t_{k}(x) are functions of x (not depending on θ), and c(θ) > 0 and w_{1}(θ),...,w_{k}(θ) are functions of the possibly vector-valued θ (not depending on x). Note that the expression for f may not be unique.

UW-Madison (Statistics)  Stat 609 Lecture 12  2014  1 / 12

beamer-tu-logo

Example 3.4.1.

To show that a family of pdf’s or pmf’s is an exponential family, we must identify the functions h(x), t_{i}(x), c(θ), and w_{i}(θ) and show that the pdf or pmf has the given form. The binomial(n,p) distribution with p ∈ (0,1) and a fixed n has pmf

\binom{n}{x} p^{x}(1 − p)^{n−x} = \binom{n}{x} (1 − p)^{n}exp( log( \frac{p}{1 − p} ) x) , x = 0,1,...,n.

Let θ = p, c(θ) = (1 − p)^{n}, w_{1}(θ) = log(\frac{p}{1−p} ), t_{1}(x) = x, and h(x) = \binom{n}{x} for x = 0,1,...,n and = 0 otherwise. Then, the binomial family with p ∈ (0,1) and a fixed n is an exponential family (k = 1). (Note that p = 0 and p = 1 are not included in the family.) Other examples: Poisson, negative binomial, normal, gamma, beta,...

Exponential families have many nice properties. The following result is useful since we can replace integration or summation by differentiation.

UW-Madison (Statistics)  Stat 609 Lecture 12  2014  2 / 12

beamer-tu-logo

Example 3.4.1.

To show that a family of pdf’s or pmf’s is an exponential family, we must identify the functions h(x), t_{i}(x), c(θ), and w_{i}(θ) and show that the pdf or pmf has the given form. The binomial(n,p) distribution with p ∈ (0,1) and a fixed n has pmf

\binom{n}{x} p^{x}(1 − p)^{n−x} = \binom{n}{x} (1 − p)^{n}exp( log( \frac{p}{1 − p} ) x) , x = 0,1,...,n.

Let θ = p, c(θ) = (1 − p)^{n}, w_{1}(θ) = log(\frac{p}{1−p} ), t_{1}(x) = x, and h(x) = \binom{n}{x} for x = 0,1,...,n and = 0 otherwise. Then, the binomial family with p ∈ (0,1) and a fixed n is an exponential family (k = 1). (Note that p = 0 and p = 1 are not included in the family.) Other examples: Poisson, negative binomial, normal, gamma, beta,...

Exponential families have many nice properties. The following result is useful since we can replace integration or summation by differentiation.

UW-Madison (Statistics)  Stat 609 Lecture 12  2014  2 / 12

beamer-tu-logo

Theorem 3.4.2.

If X has a pdf or pmf from an exponential family and w_{i}(θ)’s are differentiable functions, then

E ( ∑_{i=1}^{k} \frac{∂w_{i}(θ)}{∂θ_{j}}t_{i}(X)) = −\frac{∂ logc(θ)}{∂θ_{j}}

where θ_{j} is the jth component of θ, and

Var( ∑_{i=1}^{k} \frac{∂w_{i}(θ)}{∂θ_{j}}t_{i}(X)) = −\frac{∂^{2}logc(θ)}{∂θ_{j}^{2}} − E ( ∑_{i=1}^{k} \frac{∂^{2}w_{i}(θ)}{∂θ_{j}^{2}}t_{i}(X))

Proof.

From the exponential family expression for f_{θ}(x),

logf_{θ}(X) = logh(X) + logc(θ) + ∑_{i=1}^{k}w_{i}(θ)t_{i}(X)

Differentiating this expression leads to

\frac{∂ logf_{θ}(X)}{∂θ_{j}} = \frac{∂ logc(θ)}{∂θ_{j}} + ∑_{i=1}^{k} \frac{∂w_{i}(θ)}{∂θ_{j}}t_{i}(X)
UW-Madison (Statistics)  Stat 609 Lecture 12  2014  3 / 12

beamer-tu-logo

Taking expectation, we obtain

E ( \frac{∂ logf_{θ}(X)}{∂θ_{j}}) = \frac{∂ logc(θ)}{∂θ_{j}} + E ( ∑_{i=1}^{k} \frac{∂w_{i}(θ)}{∂θ_{j}}t_{i}(X))

If f_{θ}(x) is a pdf (the proof for pmf is similar), then the left side of the previous expression is

∫_{−∞}^{∞} \frac{∂ logf_{θ}(x)}{∂θ_{j}}f_{θ}(x)dx = ∫_{−∞}^{∞} \frac{∂f_{θ}(x)}{∂θ_{j}}dx = \frac{∂}{∂θ_{j}} ∫_{−∞}^{∞} f_{θ}(x)dx = \frac{∂1}{∂θ_{j}} = 0

We interchanged the differentiation and integration, which is justified under the exponential family assumption. This proves the first result. Note that

\frac{∂^{2}logf_{θ}(X)}{∂θ_{j}^{2}} = \frac{∂}{∂θ_{j}} [ \frac{\frac{∂f_{θ}(X)}{∂θ_{j}}}{f_{θ}(X)}] = \frac{\frac{∂^{2}f_{θ}(X)}{∂θ_{j}^{2}}}{f_{θ}(X)} − [ \frac{\frac{∂f_{θ}(X)}{∂θ_{j}}}{f_{θ}(X)}]^{2}

Then

beamer-tu-logo

E ( \frac{∂^{2}logf_{θ}(X)}{∂θ_{j}^{2}}) = ∫_{−∞}^{∞} \{ \frac{\frac{∂^{2}f_{θ}(X)}{∂θ_{j}^{2}}}{f_{θ}(X)} − [ \frac{\frac{∂f_{θ}(X)}{∂θ_{j}}}{f_{θ}(X)}]^{2}f_{θ}(x)\} dx

= ∫_{−∞}^{∞} \frac{∂^{2}f_{θ}(X)}{∂θ_{j}^{2}}dx − ∫_{−∞}^{∞} [ \frac{∂ logf_{θ}(X)}{∂θ_{j}}]^{2}f_{θ}(x)dx

= −∫_{−∞}^{∞} [ \frac{∂ logc(θ)}{∂θ_{j}} + ∑_{i=1}^{k} \frac{∂w_{i}(θ)}{∂θ_{j}}t_{i}(X)]^{2}f_{θ}(x)dx

= −Var( ∑_{i=1}^{k} \frac{∂w_{i}(θ)}{∂θ_{j}}t_{i}(X))

which follows from the first result. Then the second result follows from

\frac{∂^{2}logf_{θ}(X)}{∂θ_{j}^{2}} = \frac{∂^{2}logc(θ)}{∂θ_{j}^{2}} + ∑_{i=1}^{k} \frac{∂^{2}w_{i}(θ)}{∂θ_{j}^{2}}t_{i}(X)

beamer-tu-logo

Example 3.4.4.

If X ∼ N(μ,σ^{2}), then θ = (μ,σ) and

f_{θ}(x) = \frac{1}{\sqrt{2π}σ} exp( −\frac{(x − μ)^{2}}{2σ^{2}} ) = \frac{1}{\sqrt{2π}σ} exp( −\frac{μ^{2}}{2σ^{2}} ) exp( \frac{μx}{σ^{2}} − \frac{x^{2}}{2σ^{2}} )

Let h(x) = 1, c(θ) = \frac{1}{\sqrt{2π}σ} exp(−\frac{μ^{2}}{2σ^{2}} ), w_{1}(θ) = 1/σ^{2}, w_{2}(θ) = μ/σ^{2}, t_{1}(x) = −x^{2}/2, and t_{2}(x) = x. Then this normal family is an exponential family with k = 2. Applying Theorem 3.4.2, we obtain E(X) = μ from equation

−\frac{∂ logc(θ)}{∂μ} = \frac{μ}{σ^{2}} = E ( ∑_{i=1}^{2} \frac{∂w_{i}(θ)}{∂μ}t_{i}(X)) = E ( \frac{X}{σ^{2}} )

Also,

−\frac{∂ logc(θ)}{∂σ} = \frac{μ^{2}}{σ^{3}} + \frac{1}{σ} = E ( ∑_{i=1}^{2} \frac{∂w_{i}(θ)}{∂σ}t_{i}(X)) = E ( \frac{X^{2}}{σ^{3}} − \frac{2μX}{σ^{3}} )

Using E(X) = μ, we obtain from this equation that Var(X) = σ^{2}.

beamer-tu-logo

Natural exponential family

If η_{i} = w_{i}(θ), i = 1,...,k, and η = (η_{1},...,η_{k}), the form of f_{θ} in the exponential family becomes

f_{η}^{∗}(x) = h(x)c^{∗}(η)exp( ∑_{i=1}^{k}η_{i}t_{i}(x))

η is called the natural parameter. The set of η’s for which f_{η}^{∗}(x) is a well-defined pdf is called the natural parameter space.

Full or curved exponential families

In an exponential family, if the dimension of θ is k (there is an open set ⊂ Θ), then the family is a full exponential family. Otherwise the family is a curved exponential family.

An example of a full exponential family is N(μ,σ^{2}), μ ∈ R, σ > 0. An example of a curved exponential family is N(μ,μ^{2}), μ ∈ R.

beamer-tu-logo

Natural exponential family

If η_{i} = w_{i}(θ), i = 1,...,k, and η = (η_{1},...,η_{k}), the form of f_{θ} in the exponential family becomes

f_{η}^{∗}(x) = h(x)c^{∗}(η)exp( ∑_{i=1}^{k}η_{i}t_{i}(x))

η is called the natural parameter. The set of η’s for which f_{η}^{∗}(x) is a well-defined pdf is called the natural parameter space.

Full or curved exponential families

In an exponential family, if the dimension of θ is k (there is an open set ⊂ Θ), then the family is a full exponential family. Otherwise the family is a curved exponential family.

An example of a full exponential family is N(μ,σ^{2}), μ ∈ R, σ > 0. An example of a curved exponential family is N(μ,μ^{2}), μ ∈ R.

beamer-tu-logo

How to show a family is not an exponential family

It may be difficult to show that a family is not an exponential family. We cannot say “we are not able to express f_{θ}(x) in the form of an exponential family". If f_{θ}, θ ∈ Θ is an exponential family, then

{x : f_{θ}(x) > 0} = {x : h(x) > 0}

which does not depend on θ values. This fact can be used to show a family is non-exponential, i.e., if {x : f_{θ}(x) > 0} depends on θ, then f_{θ}, θ ∈ Θ, is not an exponential family. Consider the family of two parameters exponential distributions with pdf’s

f_{θ}(x) = \{ \begin{matrix}λ^{−1} e^{−(x−μ)/λ} & x > μ \\ 0 & x ≤ μ \end{matrix} μ ∈ R, λ > 0

It is not an exponential family because

{x : f_{θ}(x) > 0} = {x : x > μ}

UW-Madison (Statistics)  Stat 609 Lecture 12  2014  8 / 12

beamer-tu-logo

Definition 3.5.2 (location family)

Let f (x) be a given pdf. The family of pdf’s, f (x − μ), μ ∈ R, is called a location family with location parameter μ.

Examples of location families are normal, double exponential, Cauchy, logistic, and two-parameter exponential distributions with location parameter μ ∈ R and the other parameter σ or λ fixed. The pdf f (x − μ) is obtained by shifting the entire curve f (x) by an amount μ (see the figure) without changing the structure of f (x). It can be shown that X ∼ f (x − μ) iff X = Z + μ with Z ∼ f (x).

beamer-tu-logo

Definition 3.5.4 (scale family)

Let f (x) be a given pdf. The family of pdf’s, σ^{−1} f (x/σ), σ > 0, is called a scale family with scale parameter σ.

Examples of scale families are normal, double exponential, Cauchy, and logistic distributions with scale parameter σ > 0 and μ fixed, gamma(α,β) with β > 0 and α fixed, and two-parameter exponential distributions with λ > 0 and μ fixed. The pdf σ^{−1} f (x/σ) is obtained by stretching (σ > 1) or contracting (σ < 1) the curve f (x) while still maintaining the same shape. It can be shown that X ∼ σ^{−1} f (x/σ) iff X = σZ with Z ∼ f (x).

beamer-tu-logo

Definition 3.5.5 (location-scale family)

Let f (x) be a given pdf. The family of pdf’s, σ^{−1} f ((x − μ)/σ), μ ∈ R, σ > 0, is called a location-scale family with location parameter μ and scale parameter σ.

A location-scale family is a combination of a location family and a scale family: it contains a sub-family that is a location family with any fixed σ, and a sub-family that is a scale family with any fixed μ.

beamer-tu-logo

Examples of location-scale families are normal, double exponential, Cauchy, logistic, and two-parameter exponential distributions with location parameter μ ∈ R and scale parameter σ > 0. Except for the two-parameter exponential distribution, all others are symmetric about μ. If f (x) is symmetric about 0, then σ^{−1} f ((x − μ)/σ) is symmetric about μ and μ is the median of X ∼ σ^{−1} f ((x − μ)/σ); furthermore, if the expectation of f (x) exists, then μ is the expectation of σ^{−1} f ((x − μ)/σ). It can be shown that X ∼ σ^{−1} f ((x − μ)/σ) iff X = σZ + μ with Z ∼ f (x); furthermore, if E(Z^{2}) < ∞, then E(X) = σE(Z) + μ and Var(X) = σ^{2}Var(Z). The pdf f (x) in a location-scale family is standard iff the expectation ∫_{−∞}^{∞} xf (x)dx = 0 and the variance ∫_{−∞}^{∞} x^{2}f (x)dx = 1. Typically, we choose a standard f (x) to generate a location-scale family, in which case μ and σ^{2} are the expectation and variance of σ^{−1} f ((x − μ)/σ), respectively.