Sebastian Raschka
Sebastian Raschka
Assistant Professor of Statistics @ UW-Madison

STAT 479 -- Deep Learning (Spring 2019)

Table of Contents

Course Logistics




Office Hours

Course Description

Credits: 3

Course Description:

Deep learning is an exciting, young field that specializes in discovering and extracting intricate structures in large, unstructured datasets for parameterizing artificial neural networks with many layers. Since deep learning has pushed the state-of-the-art in many applications, it’s become indispensable for modern technology. This is owed to the vast utility of deep learning for tackling complex tasks in the fields of computer vision and natural language processing – tasks that humans are good at but are traditionally challenging for computers. This includes tasks such as image classification, object detection, and speech recognition.

The focus of this course will be on understanding artificial neural networks and deep learning algorithmically (discussing the math behind these methods on a basic level) and implementing network models in code as well as applying these to real-world datasets. Some of the topics that will be covered include convolutional neural networks for image classification and object detection, recurrent neural networks for modeling text, and generative adversarial networks for generating new data.

Familiarity with general machine learning concepts (such as the FS2018 STAT479: Machine Learning course) is recommended but not required. We will review some relevant background concepts, which include general machine learning concepts such as supervised learning, classification, model evaluation, etc. Furthermore, some lectures will focus on reviewing the use of Python’s stack for scientific computing (NumPy, SciPy, matplotlib) prior to the introduction of PyTorch as the main computational deep learning library that we are going to use in this course.

Learning Outcomes:

Course Prerequisites: Consent of instructor.

Course Audience: Students majoring in math or statistics or those wishing to take additional statistics courses.

Credits: 3


Deep Learning (mildly recommended)

Deep learning is a relatively young field that is advancing at a rapid pace. Unfortunately, there is no good textbook resource available for this topic. This book is an “older” book (~2014) that covers some of the topics we will discuss in class. Personally, I think this book is not ideal for teaching and probably more of a summary or reference resource.

Hence, the lecture will not be based on this book, but you may find it useful still. A free digital version shared by the authors can be found at

However, in the field of deep learning, it is highly recommended to consider reading the original papers, which I will link in this course.

PyTorch (highly recommended)

Also regarding computational technologies for deep learning, there is no good textbook resource available, yet. My deep learning background started with Theano, and I have been an avid TensorFlow user since its release in 2015. However, my own research is now more heavily focused on PyTorch these days as it is more convenient to work with (and even a tad faster on single- and multi-GPU workstations).

While there is no good textbook available on PyTorch, there is an excellent official online documentation which is the best go-to resource for PyTorch:

Illustrated Guide to Python (recommended)

This book will not be coverered in class. However, some readers asked me for good Python resources as preparation for this class, and this is one of the resources I would recommend. However, there are many other Python learning resources available online.

For instance, another great book is Allen Downey’s Think Python 2e (free PDF available at Depending on your preferred learning style, also consider learning Python interactively instead/or in addition of reading a Python book. A great interactive resource for learning Python is Codecademy: In particular, there is a free, < 10 hr interactive course:

Python’s scientific computing stack (highly recommended)

While we will be primarily focussing on PyTorch, it will be extremely convenient if you develop a basic understanding for Python’s scientific computing stack: NumPy (linear algebra library), SciPy (additional scientific functions), Matplotlib (plotting), and Pandas (data wrangling).

Class Project


The goal of working on a class project is three-fold. First, it will provide you with the opportunity to apply the concepts learned in this class creatively, which helps you with understanding material more deeply. Second, designing and working on a unique project in a team which is something that you will encounter, if you haven’t already, rather sooner than later in life, and this course project helps with preparing for that. Third, along with the opportunity to practice and the satisfaction of working creatively, students can use this project to enhance their portfolio or resume.

Note about grading

There is no “perfect project.” While you are encouraged to be ambitious, the most important aspect of this project is your learning experience. Hence, you don’t want to pick something that is too easy for you, but similarly, you don’t want to choose a project where you are not certain that is out of the scope of this class. The project proposal is not graded by how exciting your project is but based on whether you follow the objectives of the project proposal, project presentation, and project report. For instance, if your project ends up being unsuccessful – for example, if you choose to design a classifier and it doesn’t achieve the desired accuracy – it will not negatively affect your grade as long as you are honest, describe the potential issues well, and suggest improvements or further experiments. Again, the objective of this project is to provide you with hands-on practice and an opportunity to learn.

The project consists of 3 parts: a project proposal, a short project presentation, and a project report. The expectations for each part will be discussed in the following sections.

1) Project Proposal

Please note that you should use the proposal-latex file(s) for writing and submitting your proposal!

The main purpose of the project proposal is to receive feedback from the TAs/the instructor regarding whether your project is feasible and whether it is within the scope of this class. Also, the project proposal offers a chance to receive useful feedback and suggestions on your project.

For this project, you will be working in a team consisting of three students. You are encouraged to form groups by yourself as discussed in class. If you cannot find group members, the TA and I will randomly assign you to a group. If you have any concerns working with someone in your group, please talk to a TA or the instructor for accommodations.

Proposal Format:






You are expected to share the workload evenly, and every group member is expected to participate in both the experiments and writing. (As a group, you only need to submit one proposal and one report, though. So you need to work together and coordinate your efforts.)

It is crucial that you talk to each other regularly!!! Schedule regular meetings and/or use online communication tools (e.g., Gitter, Slack, or email) to stay in touch with your group members throughout the semester regarding the process of your project.

Modifications to the Proposal. After you have received feedback from the TAs/the instructor and your project proposal has been graded, you are advised to stick to the project outline in the proposal as closely as possible. However, if there is a concept introduced in a later lecture (for instance, a machine learning algorithm that you think is more appropriate then the one you proposed), you have the option to modify your proposal, but you are not penalized if you don’t. If you wish to update your project outline, talk to a TA first.

2) Project Presentation

During the last three lectures, you will be presenting your project to the class. The presentation is “free form” but should cover the following:

The presentation should be 8-10 minutes long, plus 2 minutes will be reserved for questions. All members of the group should participate in the presentation.

The voting card should be filled out as follows:

  1. Title of the Presentation, x/10, y/10, z/10
  2. Title of the Presentation, x/10, y/10, z/10 …


The awards will be computed based on the highest number of points for each category. However, one project can only receive one of the prizes.

Each of the three cards handed in will provide 3 bonus points towards your project report grade (9 pts in total).

3) Project Report

The project report is expected to be 6-8 pages long (excluding references) and should contain the follwing sections:

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Experiments
  5. Results and Discussion
  6. Conclusions
  7. Contributions

More details are provided in the LaTeX report template at

Please note that you should use the report-latex file for writing and submitting your report!

Also, you are required to submit all the code, computations, and experiments you developed and conducted for this project. Note that the quality of code will not have any influence on your grad and will merely serve as a basis to establish that the report contains original and “real” results.

Optional: Sharing your Project

You are encouraged to share your project/final project report online after you completed the course – for example, via GitHub or on a personal website online.

Below is a list of examples from last year’s Machine Learning (not deep learning) class:


The final grade will be computed using the following weighted grading scheme:

Other Important Course Information


See the Guides’s Rules, Rights and Responsibilities


By enrolling in this course, each student assumes the responsibilities of an active participant in UW-Madison’s community of scholars in which everyone’s academic work and behavior are held to the highest academic integrity standards. Academic misconduct compromises the integrity of the university. Cheating, fabrication, plagiarism, unauthorized collaboration, and helping others commit these acts are examples of academic misconduct, which can result in disciplinary action. This includes but is not limited to failure on the assignment/course, disciplinary probation, or suspension. Substantial or repeated cases of misconduct will be forwarded to the Office of Student Conduct & Community Standards for additional review. For more information, refer to


McBurney Disability Resource Center syllabus statement: “The University of Wisconsin-Madison supports the right of all enrolled students to a full and equal educational opportunity. The Americans with Disabilities Act (ADA), Wisconsin State Statute (36.12), and UW-Madison policy (Faculty Document 1071) require that students with disabilities be reasonably accommodated in instruction and campus life. Reasonable accommodations for students with disabilities is a shared faculty and student responsibility. Students are expected to inform faculty [me] of their need for instructional accommodations by the end of the third week of the semester, or as soon as possible after a disability has been incurred or recognized. Faculty [I], will work either directly with the student [you] or in coordination with the McBurney Center to identify and provide reasonable instructional accommodations. Disability information, including instructional accommodations as part of a student’s educational record, is confidential and protected under FERPA.”


Institutional statement on diversity: “Diversity is a source of strength, creativity, and innovation for UW-Madison. We value the contributions of each person and respect the profound ways their identity, culture, background, experience, status, abilities, and opinion enrich the university community. We commit ourselves to the pursuit of excellence in teaching, research, outreach, and diversity as inextricably linked goals.

The University of Wisconsin-Madison fulfills its public mission by creating a welcoming and inclusive community for people from every background – people who as students, faculty, and staff serve Wisconsin and the world.”


Note that this is a tentative schedule subject to changes.

Below is a list of topics we aim to cover. However, we will take our time, and it is more important to build a good understanding of the core concepts and the field in general rather than covering one more algorithm. Keep in mind that a good foundation will enable you to study and understand additional algorithms if the need arises.

Topics Summary


Lecture Material
Jan 23
Day 1
● Course Overview
● L01: Intro to DL
Jan 25
Day 2
L01: Intro to DL cont'd
Jan 28
Day 3
● L02: DL history
Jan 30
Day 4
Canceled due to weather-related
campus closure
Feb 01
Day 5
● L03: The Perceptron
● Start working on HW1
Due on Thu Feb 07 (11:59 pm)
Feb 04
Day 6
L03: The Perceptron cont'd
Feb 06
Day 7
● L04: Linear Algebra for Deep Learning
Feb 08
Day 8
L04: Linear Algebra for DL cont'd
Deadline for
Group Assignments
Feb 11
Day 9
● L05: Fitting Neuron Models with
   Gradient Descent
Feb 13
Day 10
L05: Fitting Neuron Models... cont'd
● Start working on HW2
Due on Thu Feb 21 (11:59 pm)
Feb 15
Day 11
● L06: Automatic Differentiation
   with PyTorch
Feb 18
Day 12
L06: Automatic Differentiation
   with PyTorch cont'd
Feb 20
Day 13
● L07: Cloud Computing
Feb 25
Day 15
HW2 discussion
L08: Logistic Reg. & Multiclass cont'd
Feb 27
Day 16
L08: Logistic Reg. & Multiclass cont'd
Mar 01
Day 17
● L09: Multilayer Perceptrons
● Start working on HW3
Due on Fri Mar 8 (11:59 pm)
Mar 04
Day 18
L09: Multilayer Perceptrons cont'd
Mar 06
Day 19
L09: Multilayer Perceptrons cont'd
Mar 08
Day 20
● L10: Regularization
Mar 11
Day 21
L10: Regularization cont'd
Mar 13
Day 22
Mar 15
Day 23
● L11: Normalization and Weight Initialization
Submit Project
Mar 18
Spring recess
Mar 20
Spring recess
Mar 22
Spring recess
Mar 25
Day 24
L11: Normalization and Weight Initialization cont'd
Mar 27
Day 25
● L12: Learning Rates and Optimization
Mar 29
Day 26
L12: Learning Rates and Optimization cont'd
Apr 01
Day 27
● L13: Intro to ConvNets (Part 1)
Apr 03
Day 28
L13: Intro to ConvNets (Part 1) cont'd
● Start working on HW4
Due on Fri, Feb 12 (11:59 pm)
Apr 05
Day 29
L13: Intro to ConvNets (Part 1) cont'd
Apr 08
Day 30
● L13: Intro to ConvNets (Part 2)
Apr 10
Day 31
L13: Intro to ConvNets (Part 2) cont'd
● L13: Intro to ConvNets (Part 3)
Apr 12
Day 32
L13: Intro to ConvNets (Part 3) cont'd
Apr 15
Day 33
● L14: Intro to RNNs (Part 1)
Apr 17
Day 34
● L14: Intro to RNNs (Part 2)
Apr 19
Day 35
● L15: Autoencoders
● L16: Variational Autoencoders
Apr 22
Day 36
● L17: Generative Adversarial Networks
Apr 24
Day 37
 Project Presentations
Apr 26
Day 38
 Project Presentations
Apr 29
Day 39
 Project Presentations
May 01
Day 40
 Project Presentations
DL Competition
Submission Deadline
Winner: Tianyu Zeng

Winning Solution
May 03
Day 41
 Project Presentations
Mon, May 06
Day 42
Final Exam
5:05 - 7:05 pm, Room: VAN VLECK B130
Final Exam
Wed, May 08
Submit Final Project Report
Submit Final Project Report

Project Presentation Awards

Without exception, we had amazing project presentations this semester. Nonetheles, we have some winners the top 5 project presentations for each of the 3 categories, as determined by voting among the ~65 students:

Best Oral Presentation:

  1. Saisharan Chimbiki, Grant Dakovich, Nick Vander Heyden (Creating Tweets inspired by Deepak Chopra), average score: 8.417

  2. Josh Duchniak, Drew Huang, Jordan Vonderwell (Predicting Blog Authors’ Age and Gender), average score: 7.663

  3. Sam Berglin, Jiahui Jiang, Zheming Lian (CNNs for 3D Image Classification), average score: 7.595

  4. Christina Gregis, Wengie Wang, Yezhou Li (Music Genre Classification Based on Lyrics), average score: 7.588

  5. Ping Yu, Ke Chen, Runfeng Yong (NLP on Amazon Fine Food Reviews) average score: 7.525

Most Creative Project:

  1. Saisharan Chimbiki, Grant Dakovich, Nick Vander Heyden (Creating Tweets inspired by Deepak Chopra), average score: 8.313

  2. Yien Xu, Boyang Wei, Jiongyi Cao (Judging a Book by its Cover: A Modern Approach), average score: 7.952

  3. Xueqian Zhang, Yuhan Meng, Yuchen Zeng (Handwritten Math Symbol Recognization), average score: 7.919

  4. Jinhyung Ahn, Jiawen Chen, Lu Li (Diagnosing Plant Diseases from Images for Improving Agricultural Food Production), average score: 7.917

  5. Poet Larsen, Reng Chiz Der, Noah Haselow (Convolutional Neural Networks for Audio Recognition), average score: 7.854

Best Visualizations:

  1. Ping Yu, Ke Chen, Runfeng Yong (NLP on Amazon Fine Food Reviews), average score: 8.189

  2. Xueqian Zhang, Yuhan Meng, Yuchen Zeng (Handwritten Math Symbol Recognization), average score: 8.153

  3. Saisharan Chimbiki, Grant Dakovich, Nick Vander Heyden (Creating Tweets inspired by Deepak Chopra), average score: 7.677

  4. Poet Larsen, Reng Chiz Der, Noah Haselow (Convolutional Neural Networks for Audio Recognition), average score: 7.656

  5. Yien Xu, Boyang Wei, Jiongyi Cao (Judging a Book by its Cover: A Modern Approach), average score: 7.490