% This is a simple LaTex sample document that gives a submission format
%   for IEEE PAMI-TC conference submissions.  Use at your own risk.

% Make two column format for LaTex 2e.\
\documentclass[11pt,twocolumn]{article} %,twocolumn

%\usepackage{times,amsmath,amsfonts}

% Use following instead for LaTex 2.09 (may need some other mods as well).
%\documentstyle[times,twocolumn]{article}
\usepackage[dvips]{graphicx,graphics}
% Set dimensions of columns, gap between columns, and paragraph indent
\setlength{\textheight}{10in} \setlength{\textwidth}{7in}
%\setlength{\columnsep}{0.3125in} \setlength{\topmargin}{0in}
\setlength{\headheight}{0in} \setlength{\headsep}{-1in}
\setlength{\parindent}{1pc}
\setlength{\oddsidemargin}{-.5in}  % Centers text.
\setlength{\evensidemargin}{-.5in}

% Add the period after section numbers.  Adjust spacing.
\newcommand{\Section}[1]{\vspace{-8pt}\section{\hskip -1em.~~#1}\vspace{-3pt}}
\newcommand{\SubSection}[1]{\vspace{-3pt}\subsection{\hskip -1em.~~#1}
        \vspace{-3pt}}
\newcommand{\bqn}{\begin{eqnarray}}
\newcommand{\eqn}{\end{eqnarray}}
\newcommand {\diff}[1] {\frac{\partial}{\partial #1}}
\newcommand{\jacob}[3]{\frac{\partial^2 #3}{\partial #1 \partial #2}}
\newcommand{\der}[2]{\frac{\partial #2}{\partial #1}}
\begin{document}

% Make title bold and 14 pt font (Latex default is non-bold, 16pt)
\title{Stat 471: Lecture 9\\
Least Squares Estimation.}
% For single author (just remove % characters)
\author{Moo K. Chung\\
mchung@stat.wisc.edu}
% For two authors (default example)
\maketitle \thispagestyle{empty}

\begin{enumerate}


\item $$\left(%
\begin{array}{cc}
  2 & 1 \\
  4 & 2 \\
\end{array}%
\right)\left(%
\begin{array}{c}
  y_1 \\
  y_2 \\
\end{array}%
\right) = \left(%
\begin{array}{c}
  1 \\
  1 \\
\end{array}%
\right).$$ The least squares solution is given by
\begin{verbatim}
>> X=[2 1
      4 2];
>> pinv(X)*[1 1]'
ans =
    0.2400
    0.1200
\end{verbatim}

\item Response function estimation. For given bivariate data
$(x_i,y_i)$. Let's see if the following model fits data.
$$Y_i=\phi(x_i) + \epsilon_i,$$
where $\epsilon_i$ is i.i.d. Gaussian noise. A more sophisticated
correlated error modeling is also possible (possible project
topic). Some possible model choice would be $\phi(x) = \beta_0 +
\beta_1 x$ (linear), $\phi(x) = \beta_0 + \beta_1 x + \beta_2 x^2$
(quadratic). In general for basis functions $\phi_i(x)$, we may let
$\phi(x) = \sum_{i=0}^m \beta_i \phi_i(x)$ and we estimate
$\beta_i$ via the least squares estimation. We need to solve the
system of the linear equation
$$ \left(%
\begin{array}{ccc}
  \phi_1(x_1)  &\cdots & \phi_m(x_1) \\
\phi_1(x_2)  &\cdots & \phi_m(x_2) \\
\cdots  & \cdots & \cdots\\
\phi_1(x_n)  &\cdots & \phi_m(x_n) \\
\end{array}\right)
\left(
\begin{array}{c}
  \beta_1 \\
  \beta_2\\
  \cdots\\
  \beta_m\\
\end{array}%
\right) = \left(%
\begin{array}{c}
  y_1 \\
  y_2\\
  \cdots\\
  y_n \\
\end{array}%
\right).$$ Then we solve the normal equation
$$X'X\beta = X'y$$
If there are more data then the basis functions, i.e. $n \geq m$,
$X'X$ is invertible. So we get
$$\hat \beta = (X'X)^{-1}X'y.$$
Popular choice for $\phi_i$ is Hermite polynomials $H_i(x)$ which
forms orthogonal polynomial basis with respect to $e^{-x^2}$ in
$\mathbb{R}$, i.e.
$$\int_{-\infty}^{\infty} H_i(x)H_j(x) e^{-x^2} \; dx =\delta_{ij}2^jj!\sqrt{\pi}.$$
The first few Hermite polynomials are $H_0(x)=1, H_1(x)=2x,
H_2(x)=4x^2-2$.

 \item Linear regression. Let $\phi(x)=\beta_0+\beta_1 x$.
Download data
http://www.stat.wisc.edu/$\sim$mchung/
teaching/data/strength.data
and save it into your $\tt{MATLAB}$ working directory or alternately
simply copy and paste into your editor.
\begin{verbatim}
load strength.data
arm=strength(:,1); grip=strength(:,2)
plot(arm,grip,'+');hold on
X=[ones(147,1) arm]
beta=pinv(X)*y
beta=inv(X'*X)*X'*y
>>beta=X\y
beta =
   16.7297
    0.5627
i=[20:200]
plot(i,beta(1) + beta(2)*i)
\end{verbatim}
\begin{figure}
\centering
\renewcommand{\baselinestretch}{1}
\includegraphics[scale=0.4]{lecture09-1.eps}
\end{figure}
\end{enumerate}
\small
Chapter 10 nonparametric regression is related to Lecture 9. Next week we will discuss about Monte-Carlo method (Chapter 6). Be prepared!
\end{document}