Assistant Professor of Statistics @ UW-Madison
Below is a list of the topics I am planning to cover in this course. Since course topics are among the most often requested information about this course, I am placing this on top of this website. More information about this course can be found in the sections that follow the course content below.
Part 1: Introduction
Part 2: Mathematical and computational foundations
Part 3: Introduction to neural networks
Part 4: Deep learning for computer vision and language modeling
Part 5: Deep generative models
Credits: 3
Course Description:
Deep learning is a field that specializes in discovering and extracting intricate structures in large, unstructured datasets for parameterizing artificial neural networks with many layers. Since deep learning has pushed the state-of-the-art in many research and application areas, it’s become indispensable for modern technology.
The focus of this course will be on understanding deep, artificial neural networks by connecting it to related concepts in statistics, such as generalized linear models and maximum likelihood estimation. Beyond covering deep learning models for predictive modeling, the latter portion of this course will focus on deep generative models and models based on stochastic variational inference, which allows for learning directed probabilistic models.
Besides covering and explaining deep learning and generative models on a mathematical and conceptual level, this course emphasizes the practical aspects of deep learning. Open-source libraries from the Python open-source ecosystem for scientific computing will be used to provide students with hands-on experience for implementing deep neural nets, working on supervised learning tasks, and applying generative models for dataset synthesis. Regarding the class project, students will form teams of three and collaboratively work on a project proposal to outline the planned scope of the project and meet with the lecturer for further discussion and feedback. After receiving feedback on the proposal, students will work independently towards the final project report, which will be submitted in the form of a conference paper for peer-review by other students and the lecturer. Finally, the students will give an 8-10-minute talk at the end of the semester to formally present their projects in class.
Learning Outcomes:
Course Rerequisites:
Along with introducing of the concepts of deep learning and generative models, the in-class lectures will provide a refresher on relevant concepts from calculus and linear algebra; however, a calculus background (e.g., Math 221) and a linear algebra background (e.g., Math 340) is recommended. While this course will also provide an introduction to the basics of the Python programming language for machine learning, it is highly recommended that students are familiar with basic programming and have completed an introductory programming class.
The official requisites are
Course Audience:
Students majoring in math or statistics or those wishing to take additional statistics courses.
During this online semester, we will mainly be using Canvas for this course so that everything is conveniently organized in one place. This includes the course material, announcements, homework submissions, exam, and discussions. I highly recommend using Firefox or Chrome for using Canvas because it seems some features are not well supported in Safari, yet.
For personal questions (course accommodations due to illness, missed assignments, etc.), please choose the option “This private post is only visible to Instructors” before posting.
Lecture material will usually be posted every Tuesday and Thursday morning.
Online.
Prof. Sebastian Raschka (Instructor)
Zhongjie Yu (Teaching Assistant)
Regarding the use of Python in this course, we will mainly focus on two libraries: NumPy and PyTorch*. You can think of NumPy as a linear algebra library that provides utilities similar to MatLab (if you are familiar with MatLab). It’s a library that is used in almost any scientific computing task and other libraries in Python and is generally useful. PyTorch is the main deep learning library we will be using.
In any case, you don’t need to be an expert Python programmer to follow the content in this class, (and I will teach you about PyTorch in this course, so no worries about learning it beforehand). However, some basic familiarity with Python will be necessary in order to use these libraries.
Below are two beginner-friendly resources that I think will be useful to learning the Python basics:
Python for Beginners (A collection of short video Lectures)
A great video series by educators at Microsoft, which was recently made available for free on YouTube: Python for Beginners.
Python Like You Mean It
A more comprehensive free introduction for getting started with Python and its main scientific computing libraries: https://www.pythonlikeyoumeanit.com.
We won’t be using NumPy (a linear algebra and array library) extensively in this course. However, the NumPy API is very similar PyTorch’s tensor API, so understanding NumPy will be very useful for understanding and working with PyTorch. Below is tutorial that I wrote based on my lecture notes. I also added videos from last semester’s Stat 451: Machine Learning:
Scientific Computing in Python: Introduction to NumPy and Matplotlib – Including Video Tutorials.
Deep learning is a new and very fast moving field, and many of the knowledge is contained in freely available research articles and other articles shared freely on the internet. As of today, there is also no nice textbook available that would suit the structure of this course.
Thus, I will link free resources, including internet articles and research articles that are relevant for the course. The book suggestions are recommendations but not requirements. I will not use any chapters directly for this course, but you can use them as a personal reference.
Deep Learning with PyTorch
The “Deep Learning with PyTorch” offers a nice overview for beginners. I have written a more detailed review here.
A link direct link to the free PDF version is available at: https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf. A print version can be obtained from https://www.manning.com/books/deep-learning-with-pytorch.
Deep Learning – Goodfellow, Bengio, Courville
The “Deep Learning” book by Goodfellow et al. is nice for deeper theoretical coverage of deep learning. The book is also officially and freely available as web version at http://www.deeplearningbook.org
Python Machine Learning, 3rd Edition – Raschka & Mirjalili
My Python Machine Learning book provides intro to general machine learning that could be useful as background information on the general concepts. The deep learning chapters are in TensorFlow though, and we will be using PyTorch in this class. However, the explanations of convolutional networks, recurrent neural networks, etc. are still useful, and I will be using several figures from this book.
The final grade will be computed using the following weighted grading scheme:
The final letter grade will be based on the percent of the total points accumulated in the course. The proposed grade cut-offs are as follows:
However, please note that the grades are subject to curving and the cut-offs may be adjusted such that a grade distribution similar to previous semester can be achieved.
The midterm exam will be the only exam in this class. There is no final exam. The midterm exam will take place on Tuesday, March 30th, 6:00 - 7:30 pm. Please block off this time on your calendar.
Please note that the exam is taking place in the evening to accomodate for different time zones. More details about the exam (format and types of questions) will be shared at a later time in this semester.
The goal of working on a class project is three-fold. First, it will provide you with the opportunity to apply the concepts learned in this class creatively, which helps you with understanding material more deeply. Second, designing and working on a unique project in a team which is something that you will encounter, if you haven’t already, rather sooner than later in life, and this course project helps with preparing for that. Third, along with the opportunity to practice and the satisfaction of working creatively, students can use this project to enhance their portfolio or resume.
The project consists of 3 parts:
For the class project, you will be working in teams of three.
The project proposal is a short 2-3 page report outlining your plans; it will be due on March 1st. The goal of this proposal is to share your plans with me (the instructor) so that I can provide constructive feedback.
The project presentation will be a (pre-recorded) oral presentation (approx. 8-10 min) that is due at the end of the semester. The goal of this presentation is to practice summarizing and communicating your project to an audience.
The final report is a 8-page report in a conference paper format. It will be submitted at the end of the semester. The purpose of this report is to provide a comprehensive and professional description of your project.
As part of this project experience, each student will also be peer-reviewing 3 talks and 3 reports. More details about the report format and procedures will be shared later in this semester. Examples will be provided.
See the Guides’s Rules, Rights and Responsibilities
By enrolling in this course, each student assumes the responsibilities of an active participant in UW-Madison’s community of scholars in which everyone’s academic work and behavior are held to the highest academic integrity standards. Academic misconduct compromises the integrity of the university. Cheating, fabrication, plagiarism, unauthorized collaboration, and helping others commit these acts are examples of academic misconduct, which can result in disciplinary action. This includes but is not limited to failure on the assignment/course, disciplinary probation, or suspension. Substantial or repeated cases of misconduct will be forwarded to the Office of Student Conduct & Community Standards for additional review. For more information, refer to studentconduct.wiscweb.wisc.edu/academic-integrity/.
McBurney Disability Resource Center syllabus statement: “The University of Wisconsin-Madison supports the right of all enrolled students to a full and equal educational opportunity. The Americans with Disabilities Act (ADA), Wisconsin State Statute (36.12), and UW-Madison policy (Faculty Document 1071) require that students with disabilities be reasonably accommodated in instruction and campus life. Reasonable accommodations for students with disabilities is a shared faculty and student responsibility. Students are expected to inform faculty [me] of their need for instructional accommodations by the end of the third week of the semester, or as soon as possible after a disability has been incurred or recognized. Faculty [I], will work either directly with the student [you] or in coordination with the McBurney Center to identify and provide reasonable instructional accommodations. Disability information, including instructional accommodations as part of a student’s educational record, is confidential and protected under FERPA.” http://mcburney.wisc.edu/facstaffother/faculty/syllabus.php
Institutional statement on diversity: “Diversity is a source of strength, creativity, and innovation for UW-Madison. We value the contributions of each person and respect the profound ways their identity, culture, background, experience, status, abilities, and opinion enrich the university community. We commit ourselves to the pursuit of excellence in teaching, research, outreach, and diversity as inextricably linked goals.
The University of Wisconsin-Madison fulfills its public mission by creating a welcoming and inclusive community for people from every background – people who as students, faculty, and staff serve Wisconsin and the world.” https://diversity.wisc.edu/
During the global COVID-19 pandemic, we must prioritize our collective health and safety to keep ourselves, our campus, and our community safe. As a university community, we must work together to prevent the spread of the virus and to promote the collective health and welfare of our campus and surrounding community.
Information on COVID-19 is constantly changing. Students should be attentive to University communications regarding COVID-19 that may alter instruction and supersede parts of this syllabus.
Please see https://covidresponse.wisc.edu for the recent updates and developments.