% This is a simple LaTex sample document that gives a submission format
%   for IEEE PAMI-TC conference submissions.  Use at your own risk.

% Make two column format for LaTex 2e.
\documentclass[10pt,twocolumn]{article} %,twocolumn
\usepackage[dvips]{graphicx,graphics}
\usepackage{times,amsmath,amsfonts}

% Use following instead for LaTex 2.09 (may need some other mods as well).
% \documentstyle[times,twocolumn]{article}

% Set dimensions of columns, gap between columns, and paragraph indent
\setlength{\textheight}{9.9in} \setlength{\textwidth}{6.9in}
%\setlength{\columnsep}{0.3125in} \setlength{\topmargin}{0in}
\setlength{\headheight}{0in} \setlength{\headsep}{-1in}
\setlength{\parindent}{1pc}
\setlength{\oddsidemargin}{-.2in}  % Centers text.
\setlength{\evensidemargin}{-.2in}

% Add the period after section numbers.  Adjust spacing.
\newcommand{\Section}[1]{\vspace{-8pt}\section{\hskip -1em.~~#1}\vspace{-3pt}}
\newcommand{\SubSection}[1]{\vspace{-3pt}\subsection{\hskip -1em.~~#1}
        \vspace{-3pt}}
\newcommand{\bqn}{\begin{eqnarray}}
\newcommand{\eqn}{\end{eqnarray}}
\newcommand{\diff}[1] {\frac{\partial}{\partial #1}}
\newcommand{\jacob}[3]{\frac{\partial^2 #3}{\partial #1 \partial #2}}
\newcommand{\der}[2]{\frac{\partial #2}{\partial #1}}
\begin{document}

% Don't want date printed
%\date{January 23, 2003}

% Make title bold and 14 pt font (Latex default is non-bold, 16pt)
\title{Stat 312: Lecture 19\\
Least squares estimation}

% For single author (just remove % characters)
\author{Moo K. Chung\\
mchung@stat.wisc.edu}
\date{Nov 30, 2004}
% For two authors (default example)
\maketitle \thispagestyle{empty} \subsection*{Concepts}
\begin{enumerate}

\item {\em Least squares method.} In the previous lecture, we studied the least squares
estimation method for estimating parameters in linear regression.
This method can be used to estimate other parameters in a different
model. Given measurements $y_1, y_2,\cdots, y_n$, they can be
modeled as $$Y_i = \mu + \epsilon_i,$$ where $\mathbb{E} \epsilon_i
= 0$ and $\mathbb{V} \epsilon_i =\sigma^2$ with no assumption of
normality. We are interested in estimating $\mu=\mathbb{E} Y$, the
population mean. Let $\hat \mu$ be the estimator of $\mu$. Then the
predicted value is $\hat y_i = \hat \mu$ and the residual error is
$r_i = y_i - \hat y_i$. So the total sum of the squared errors (SSE)
is
$$SSE = \sum_{i=1}^n (y_i - \hat \mu)^2.$$
To find minimum of SSE, we differentiate $SSE$ with respect to $\hat
\mu$ and get $\hat \mu = \bar y$.

\item {\em Weighted least squares method.} Suppose we have two population. Measurement are taken from the
first population: $x_1,x_2,\cdots,x_m$ and the second population:
$y_1,y_2,\cdots, y_n$. They are modeled as
$$X_i = \mu + \epsilon_i$$ with $\mathbb{E} \epsilon_i = 0$ and
$\mathbb{V} \epsilon_i = \sigma_1$ while
$$Y_i = \mu + e_i$$ with $\mathbb{E} e_i = 0$ and
$\mathbb{V} e_i = \sigma_2$. Errors $\epsilon_i$ and $e_j$ are
assumed to be independent. We can again use the least squares method
to estimate the common population mean $\mu$ but this problem is
more complicated than the previous case due to the different
variabilities. We solve this problem by transforming the above
equations to have the same variabilities:
$$\frac{X_i}{\sigma_1} = \frac{\mu}{\sigma_1} +
\frac{\epsilon_1}{\sigma_1},$$
$$\frac{Y_i}{\sigma_2} = \frac{\mu}{\sigma_2} +
\frac{e_1}{\sigma_2}.$$ Then $\frac{\epsilon_1}{\sigma_1}$ and
$\frac{e_1}{\sigma_a2}$ are i.i.d and SSE is given by

$$SSE = \frac{1}{\sigma_1^2}\sum_{i=1}^m (x_i - \hat \mu)^2+\frac{1}{\sigma_2^2}\sum_{i=1}^n (y_i - \hat \mu)^2.$$
We again differentiate SSE with respect to $\hat \mu$. The
nontrivial solution is $$\hat \mu =
\frac{1}{m/\sigma_1^2+n/\sigma_2^2}\Big[ \frac{m}{\sigma_1^2} \bar x
+ \frac{n}{\sigma_2^2}\bar y\Big].$$ This method is called the {\em
weighted least squares} method.

\end{enumerate}

{\em Example} Line probe measurements of mice micro-CT breast cancer
imaging. Download data files from
{\tt http://www.stat.wisc.edu/$\sim$mchung/\\
teaching/data/lineprobe/} into your computer directory.
\begin{verbatim}
> a<-read.table("d:/stat312/probe1.txt
",header=T)
> a
    probe1
1  -403.30
2  -379.64
3  -359.26 ...
>probe1 <-a$probe1
> probe1
 [1] -403.30 -379.64 -359.26 ...
> mean(probe1)
[1] -389.8323
> var(probe1)
[1] 1043.839
> mean(probe2)
[1] -388.046
> var(probe2)
[1] 812.3707
> mean(probe3)
[1] -385.6897
> var(probe3)
[1] 796.193
> mean(probe4)
[1] -437.0326
> var(probe4)
[1] 382.7925
\end{verbatim}
For in this example, the least squares estimation for the population
mean is $-404.7$.

{\bf Homework 6.} Due Dec 14 9:30AM. Solutions will be handed out
exactly at 9:30AM. So no late homework will be accepted. 8.52, 9.2,
9.30, 9.36, 9.48, 12.16, 12.26.





\end{document}