beamer-tu-logo

Lecture 15: Covariance and correlation coefficient

Definition 4.5.1.

The covariance of random variables X and Y is defined as

Cov(X,Y ) = E[(X − E(X)][Y − E(Y )]

provided that the expectation exists.

Definition 4.5.2.

The correlation (coefficient) of random variables X and Y is defined as ρ_{X,Y} = \frac{Cov(X,Y )}{\sqrt{Var(X)Var(Y )}}

Theorem 4.5.3.

For any random variables X and Y ,

Cov(X,Y ) = E(XY ) − E(X)E(Y ). By Cauchy-Schwartz’s inequality, [Cov(X,Y )]^{2} ≤ Var(X)Var(Y ) and, hence, |ρ_{X,Y}| ≤ 1.

UW-Madison (Statistics)  Stat 609 Lecture 15  2014  1 / 12

beamer-tu-logo

Lecture 15: Covariance and correlation coefficient

Definition 4.5.1.

The covariance of random variables X and Y is defined as

Cov(X,Y ) = E[(X − E(X)][Y − E(Y )]

provided that the expectation exists.

Definition 4.5.2.

The correlation (coefficient) of random variables X and Y is defined as ρ_{X,Y} = \frac{Cov(X,Y )}{\sqrt{Var(X)Var(Y )}}

Theorem 4.5.3.

For any random variables X and Y ,

Cov(X,Y ) = E(XY ) − E(X)E(Y ). By Cauchy-Schwartz’s inequality, [Cov(X,Y )]^{2} ≤ Var(X)Var(Y ) and, hence, |ρ_{X,Y}| ≤ 1.

UW-Madison (Statistics)  Stat 609 Lecture 15  2014  1 / 12

beamer-tu-logo

Lecture 15: Covariance and correlation coefficient

Definition 4.5.1.

The covariance of random variables X and Y is defined as

Cov(X,Y ) = E[(X − E(X)][Y − E(Y )]

provided that the expectation exists.

Definition 4.5.2.

The correlation (coefficient) of random variables X and Y is defined as ρ_{X,Y} = \frac{Cov(X,Y )}{\sqrt{Var(X)Var(Y )}}

Theorem 4.5.3.

For any random variables X and Y ,

Cov(X,Y ) = E(XY ) − E(X)E(Y ). By Cauchy-Schwartz’s inequality, [Cov(X,Y )]^{2} ≤ Var(X)Var(Y ) and, hence, |ρ_{X,Y}| ≤ 1.

UW-Madison (Statistics)  Stat 609 Lecture 15  2014  1 / 12

beamer-tu-logo

If large values of X tend to be observed with large (or small) values of Y and small values of X with small (or large) values of Y , then Cov(X,Y ) > 0 (or < 0). If Cov(X,Y ) = 0, then we say that X and Y are uncorrelated. The correlation is a standardized value of the covariance.

Theorem 4.5.6.

If X and Y are random variables and a and b are constants, then Var(aX + bY ) = a^{2}Var(X) + b^{2}Var(Y ) + 2abCov(X,Y )

Theorem 4.5.6 with a = b = 1 implies that, if X and Y are positively correlated, then the variation in X + Y is greater than the sum of the variations in X and Y ; but if they are negatively correlated, then the variation in X + Y is less than the sum of the variations. This result is useful in statistical applications.

Example 4.5.4.

The joint pdf of (X,Y ) is

beamer-tu-logo

If large values of X tend to be observed with large (or small) values of Y and small values of X with small (or large) values of Y , then Cov(X,Y ) > 0 (or < 0). If Cov(X,Y ) = 0, then we say that X and Y are uncorrelated. The correlation is a standardized value of the covariance.

Theorem 4.5.6.

If X and Y are random variables and a and b are constants, then Var(aX + bY ) = a^{2}Var(X) + b^{2}Var(Y ) + 2abCov(X,Y )

Theorem 4.5.6 with a = b = 1 implies that, if X and Y are positively correlated, then the variation in X + Y is greater than the sum of the variations in X and Y ; but if they are negatively correlated, then the variation in X + Y is less than the sum of the variations. This result is useful in statistical applications.

Example 4.5.4.

The joint pdf of (X,Y ) is

beamer-tu-logo

If large values of X tend to be observed with large (or small) values of Y and small values of X with small (or large) values of Y , then Cov(X,Y ) > 0 (or < 0). If Cov(X,Y ) = 0, then we say that X and Y are uncorrelated. The correlation is a standardized value of the covariance.

Theorem 4.5.6.

If X and Y are random variables and a and b are constants, then Var(aX + bY ) = a^{2}Var(X) + b^{2}Var(Y ) + 2abCov(X,Y )

Theorem 4.5.6 with a = b = 1 implies that, if X and Y are positively correlated, then the variation in X + Y is greater than the sum of the variations in X and Y ; but if they are negatively correlated, then the variation in X + Y is less than the sum of the variations. This result is useful in statistical applications.

Example 4.5.4.

The joint pdf of (X,Y ) is

beamer-tu-logo

The marginal of X is uniform(0,1), since for 0 < x < 1,

f_{X}(x) = ∫_{−∞}^{∞} f (x,y)dy = ∫_{x}^{x+1}dy = x + 1 − x = 1

For 0 < y < 1,

f_{Y}(y) = ∫_{−∞}^{∞} f (x,y)dx = ∫_{0}^{y} dx = y − 0 = y

and for 1 ≤ y < 2,

f_{Y}(y) = ∫_{−∞}^{∞} f (x,y)dx = ∫_{y−1}^{1} dx = 1 − (y − 1) = 2 − y

i.e.,

f_{Y}(y) = \{ \begin{matrix}2 − y & 1 ≤ y < 2 \\ y & 0 < y < 1 \\ 0 & otherwise \end{matrix}

Thus, E(X) = 1/2 and Var(X) = 1/12, and

E(Y ) = ∫_{0}^{1}y^{2}dy + ∫_{1}^{2}y(2 − y)dy = \frac{1}{3} + 4 − 1 − \frac{1}{3}(8 − 1) = 1

Var(Y ) = E(Y^{2})−1=∫_{0}^{1}y^{3}dy +∫_{1}^{2}y^{2}(2−y)dy −1 = \frac{1}{4} + \frac{14}{3} − \frac{15}{4} −1 = \frac{1}{6}

UW-Madison (Statistics)  Stat 609 Lecture 15  2014  3 / 12

beamer-tu-logo

Also,

E(XY ) = ∫_{0}^{1}∫_{x}^{x+1}xydydx = ∫_{0}^{1}( x^{2} + \frac{x}{2}) dx = \frac{1}{3} + \frac{1}{4} = \frac{7}{12}

Hence, Cov(X,Y ) = E(XY ) − E(X)E(Y ) = \frac{7}{12} − \frac{1}{2} = \frac{1}{12}

ρ_{X,Y} = \frac{Cov(X,Y )}{\sqrt{Var(X)Var(Y )}}= \frac{1/12}{\sqrt{1/12 × 1/6}} \frac{1}{\sqrt{2}}

Theorem 4.5.7.

For random variables X and Y , |ρ_{X,Y}| = 1 iff P(Y = aX + b) = 1 for constants a and b, where a > 0 if ρ_{X,Y} = 1 and a < 0 if ρ_{X,Y} = −1.

The proof of this theorem is actually discussed when we study Cauchy-Schwartz’s inequality (when the equality holds). If there is a line, y = ax + b with a ≠ 0, such that the values of the 2-dimensional random vector (X,Y ) have a high probability of being near this line, then the correlation between X and Y will be near 1 or −1.

UW-Madison (Statistics)  Stat 609 Lecture 15  2014  4 / 12

beamer-tu-logo

E(XY ) = ∫_{0}^{1}∫_{x}^{x+1}xydydx = ∫_{0}^{1}( x^{2} + \frac{x}{2}) dx = \frac{1}{3} + \frac{1}{4} = \frac{7}{12}

Hence, Cov(X,Y ) = E(XY ) − E(X)E(Y ) = \frac{7}{12} − \frac{1}{2} = \frac{1}{12}

ρ_{X,Y} = \frac{Cov(X,Y )}{\sqrt{Var(X)Var(Y )}}= \frac{1/12}{\sqrt{1/12 × 1/6}} \frac{1}{\sqrt{2}}

Theorem 4.5.7.

For random variables X and Y , |ρ_{X,Y}| = 1 iff P(Y = aX + b) = 1 for constants a and b, where a > 0 if ρ_{X,Y} = 1 and a < 0 if ρ_{X,Y} = −1.

The proof of this theorem is actually discussed when we study Cauchy-Schwartz’s inequality (when the equality holds). If there is a line, y = ax + b with a ≠ 0, such that the values of the 2-dimensional random vector (X,Y ) have a high probability of being near this line, then the correlation between X and Y will be near 1 or −1.

UW-Madison (Statistics)  Stat 609 Lecture 15  2014  4 / 12

beamer-tu-logo

On the other hand, X and Y may be highly related but have no linear relationship, and the correlation could be near 0.

Example 4.5.8, 4.5.9.

Consider (X,Y ) having pdf

f (x,y) = \{ \begin{matrix}10 & 0 < x < 1, x < y < x + 0.1 \\ 0 & otherwise \end{matrix}

This is the same as the pdf in Example 4.5.4 except that x < y < x + 0.1 (instead of x < y < x + 1) in the region {f (x,y) > 0}. The same calculation as in Example 4.5.4 shows that ρ_{X,Y} = \sqrt{100/101} , which is almost 1.

beamer-tu-logo

On the other hand, X and Y may be highly related but have no linear relationship, and the correlation could be near 0.

Example 4.5.8, 4.5.9.

Consider (X,Y ) having pdf

f (x,y) = \{ \begin{matrix}10 & 0 < x < 1, x < y < x + 0.1 \\ 0 & otherwise \end{matrix}

This is the same as the pdf in Example 4.5.4 except that x < y < x + 0.1 (instead of x < y < x + 1) in the region {f (x,y) > 0}. The same calculation as in Example 4.5.4 shows that ρ_{X,Y} = \sqrt{100/101} , which is almost 1.

beamer-tu-logo

Consider (X,Y ) having a different pdf

f (x,y) = \{ \begin{matrix}5 & −1 < x < 1, x^{2} < y < x^{2} + 0.1 \\ 0 & otherwise \end{matrix}

By symmetry, E(X) = 0 and E(XY ) = 0. Hence, Cov(X,Y ) = ρ_{X,Y} = 0. There is actually a strong relationship between X and Y . But this relationship is not linear. The correlation does not measure any nonlinear relationship.

Example 4.5.10 (bivariate normal distribution).

Consider (X,Y ) ∼ the bivariate normal distribution pdf on R^{2}:

f (x,y) = \frac{exp( −\frac{(x−μ_{1})^{2}}{2σ_{1}^{2}(1−ρ^{2})} + \frac{ρ(x−μ_{1})(y−μ_{2})}{σ_{1}σ_{2}(1−ρ^{2})} − \frac{(y−μ_{2})^{2}}{2σ_{2}^{2}(1−ρ^{2})} )}{2πσ_{1}σ_{2}\sqrt{1 − ρ^{2}}} (x,y) ∈ R^{2}

where μ_{1} ∈ R, μ_{2} ∈ R, σ_{1} > 0, σ_{2} > 0, and −1 < ρ < 1 are constants. This 2-dimensional pdf has a very nice shape, as the following figure shows.

UW-Madison (Statistics)  Stat 609 Lecture 15  2014  6 / 12

beamer-tu-logo

f (x,y) = \{ \begin{matrix}5 & −1 < x < 1, x^{2} < y < x^{2} + 0.1 \\ 0 & otherwise \end{matrix}

By symmetry, E(X) = 0 and E(XY ) = 0. Hence, Cov(X,Y ) = ρ_{X,Y} = 0. There is actually a strong relationship between X and Y . But this relationship is not linear. The correlation does not measure any nonlinear relationship.

Example 4.5.10 (bivariate normal distribution).

Consider (X,Y ) ∼ the bivariate normal distribution pdf on R^{2}:

f (x,y) = \frac{exp( −\frac{(x−μ_{1})^{2}}{2σ_{1}^{2}(1−ρ^{2})} + \frac{ρ(x−μ_{1})(y−μ_{2})}{σ_{1}σ_{2}(1−ρ^{2})} − \frac{(y−μ_{2})^{2}}{2σ_{2}^{2}(1−ρ^{2})} )}{2πσ_{1}σ_{2}\sqrt{1 − ρ^{2}}} (x,y) ∈ R^{2}

where μ_{1} ∈ R, μ_{2} ∈ R, σ_{1} > 0, σ_{2} > 0, and −1 < ρ < 1 are constants. This 2-dimensional pdf has a very nice shape, as the following figure shows.

UW-Madison (Statistics)  Stat 609 Lecture 15  2014  6 / 12

beamer-tu-logo

What is the marginal pdf of X? By definition, the marginal pdf is

f_{X}(x) = ∫_{−∞}^{∞} f (x,y)dy

By taking terms that are related to x only out of the integral, we obtain that f_{X}(x) is equal to

beamer-tu-logo

\frac{exp( −\frac{(x−μ_{1})^{2}}{2σ_{1}^{2}(1−ρ^{2})} )}{2πσ_{1}σ_{2}\sqrt{1 − ρ^{2}}} ∫_{−∞}^{∞} exp( \frac{ρ(x − μ_{1})(y − μ_{2})}{σ_{1}σ_{2}(1 − ρ^{2})} − \frac{(y − μ_{2})^{2}}{2σ_{2}^{2}(1 − ρ^{2})} ) dy

= \frac{exp( −\frac{(x−μ_{1})^{2}}{2σ_{1}^{2}} )}{2πσ_{1}σ_{2}\sqrt{1 − ρ^{2}}} ∫_{−∞}^{∞} exp( −\frac{[(y − μ_{2}) − \frac{ρσ_{2}}{σ_{1}} (x − μ_{1})]^{2}}{2σ_{2}^{2}(1 − ρ^{2})} ) dy
= \frac{1}{\sqrt{2π}σ_{1}} exp( −\frac{(x − μ_{1})^{2}}{2σ_{1}^{2}} )

which is the pdf of N(μ_{1},σ_{1}^{2}) (as expected). Similarly, the marginal pdf of Y is that of N(μ_{2},σ_{2}^{2}). We now calculate the correlation coefficient

ρ_{X,Y} = \frac{Cov(X,Y )}{\sqrt{Var(X)Var(Y )}}= E ( \frac{X − μ_{1}}{σ_{1}} ) ( \frac{Y − μ_{2}}{σ_{2}} )
= ∫_{−∞}^{∞} ∫_{−∞}^{∞} ( \frac{x − μ_{1}}{σ_{1}} ) ( \frac{y − μ_{2}}{σ_{2}} ) f (x,y)dxdy

UW-Madison (Statistics)  Stat 609 Lecture 15  2014  8 / 12

beamer-tu-logo

Letting s = ( \frac{x−μ_{1}}{σ_{1}} ) ( \frac{y−μ_{2}}{σ_{2}} ) and t = \frac{x−μ_{1}}{σ_{1}} , we obtain

ρ_{X,Y} = ∫_{−∞}^{∞} ∫_{−∞}^{∞} sf ( σ_{1}t + μ_{1},σ_{2} \frac{s}{t} + μ_{2}) \frac{σ_{1}σ_{2}}{|t|} dsdt

= ∫_{−∞}^{∞} ∫_{−∞}^{∞} \frac{s exp( −\frac{t^{2}−2ρs+s^{2}/t^{2}}{2(1−ρ^{2})} )}{2πσ_{1}σ_{2}\sqrt{1 − ρ^{2}}} \frac{σ_{1}σ_{2}}{|t|} dsdt
= ∫_{−∞}^{∞} \frac{exp(−t^{2}/2)}{2π\sqrt{(1 − ρ^{2})t^{2}}} [∫_{−∞}^{∞} s exp( −\frac{(s − ρt^{2})^{2}}{2(1 − ρ^{2})t^{2}} ) ds] dt
= ∫_{−∞}^{∞} \frac{exp(−t^{2}/2)}{\sqrt{2π}} ρt^{2}dt = \frac{ρ}{\sqrt{2π}} ∫_{−∞}^{∞} t^{2}exp(−t^{2}/2)dt
= ρ

Thus, all five parameters in the bivariate normal distribution have meanings: μ_{1} and μ_{2} are two marginal means. σ_{1}^{2} and σ_{2}^{2} are two marginal variances. ρ is the correlation coefficient for the two marginal variables.

UW-Madison (Statistics)  Stat 609 Lecture 15  2014  9 / 12

beamer-tu-logo

Theorem 4.7.9 (Covariance inequality)

Let X be a random variable and g and h be nondecreasing functions such that E[g(X)], E[h(X)], and E[g(X)h(X)] exist. Then

Cov(g(X),h(X)) ≥ 0.

Proof.

Let Z = h(X), μ_{z} = E[h(X)], and h^{−1} (t) = inf{x : h(x) ≥ t}. Then h(x) ≥ μ_{z} iff x ≥ h^{−1} (μ_{z}), and

Cov(g(X),h(X))

= E[{g(X) − E[g(X)]}(Z − μ_{z})] = E[g(X)(Z − μ_{z})]
= E[g(X)(Z − μ_{z})I({Z < μ_{z}})] + E[g(X)(Z − μ_{z})I({Z ≥ μ_{z}})]
≥ E[g(h^{−1} (μ_{z}))(Z − μ_{z})I({Z < μ_{z}})]

+E[g(h^{−1} (μ_{z}))(Z − μ_{z})I({Z ≥ μ_{z}})]

= E[g(h^{−1} (μ_{z}))(Z − μ_{z})] = g(h^{−1} (μ_{z}))E(Z − μ_{z})
= 0
UW-Madison (Statistics)  Stat 609 Lecture 15  2014  10 / 12

beamer-tu-logo

Multivariate expectation

The expectation of a random vector X = (X_{1},...,X_{n}) is defined as E(X) = (E(X_{1}),...,E(X_{n})), provided that E(X_{i}) exists for any i. When M is a matrix whose (i,j)th element if a random variable X_{ij}, E(M) is defined as the matrix whose (i,j)th element is E(X_{ij}), provided that E(X_{ij}) exists for any (i,j).

Variance-covariance matrix

The concept of mean and variance can be extended to random vectors: for an n-dimensional random vector X = (X_{1},...,X_{n}), its mean is E(X) and its variance-covariance matrix is

Var(X) = E{[X − E(X)][X − E(X)]′}
= E(XX′) − E(X)E(X′)

which is an n × n symmetric matrix whose ith diagonal element is the variance Var(X_{i}) and (i,j)th off-diagonal element is the covariance Cov(X_{i},X_{j}).

beamer-tu-logo

Let c ∈ R^{n} be a constant vector. By the linearity of integrals,

E(c′X) = c′E(X)
Var(c′X) = E[c′X − c′E(X)]^{2}
= E{c′[(X − E(X)][X − E(X)]′c}
= c′E{[X − E(X)][X − E(X)]′}c
= c′Var(X)c

Since Var(c′X) ≥ 0, c′Var(X)c ≥ 0 for any c, which means that the matrix Var(X) is nonnegative definite. If the rank of Var(X) is n, then this matrix is positive definite and its inverse exists. If the rank of Var(X) is r < n, then, with probability equal to 1, X is in a subspace of R^{n} with dimension r. If A is a constant m × n matrix, then

E(AX) = AE(X)  and  Var(AX) = AVar(X)A′
UW-Madison (Statistics)  Stat 609 Lecture 15  2014  12 / 12