Syllabus

Getting Started

Please read through this document for the course. It is not that long, and it will save us all time if you know this information about the course.

New Students

For students who joined after the first lecture: if you have recently registered, it is your responsibility to catch up. Course information, the syllabus, the eClass site, and the Piazza discussion forum are all linked here. This course website contains all lecture information and the eClass site contains all coursework required. Read the lecture slides and all course information to catch up. Your questions about the course logistics are likely answered in the information provided, so please look there first. You can come to the TA or instructor office hours if you have specific questions about something in the lectures.

Class Times:

Tuesdays and Thursdays, 2:00 pm–3:20 pm

Location:

TEL 150

Instructor:

Nidhi Hegde (nidhih at ualberta.ca)

Office Hours:

Fridays 1:00 pm - 2:00 pm in ATH 3-04

TA Email

cmput267@ualberta.ca

eClass

https://eclass.srv.ualberta.ca/course/view.php?id=95783

Syllabus

A pdf including the same information as below.

Textbook

The Machine Learning Notes.

TAs and Office Hours

There will be TA office hours both in-person and online. You can sign up for 10-minute slots, using this excel sheet.

TA	Day and Time	Location
Shreya Pekhale	Mondays 9:00 am - 10:00 am	Google Meet
Aniket Sharma	Mondays 3:00 pm - 4:00 pm	CAB 313
Olya Mastikhina	Tuesdays 11:00 am - 12:00 pm	CAB 313
Kailash Seshadri	Tuesdays 12:00 pm - 1:00 pm	Google Meet
Vlad Tkachuk	Wednesdays 2:00 pm - 3:00 pm	CAB 313
Vedd Kuknur	Wednesdays 4:00 pm - 5:00pm	CAB 313
Duc Thang Chu	Thursdays 10:00 am - 11:00 pm	CAB 313
Guoqing Luo	Thursdays 4:00 pm - 5:00 pm	Google Meet
Yongchang Hao	Fridays 2:00 pm - 3:00pm	Google Meet

Contact Information: cmput267@ualberta.ca

Course Objective

The field of machine learning involves the development of statistical algorithms that can learn from data, and make predictions on data. These algorithms and concepts are used in a range of computing disciplines, including artificial intelligence, robotics, computer vision, natural language processing, data mining, information retrieval, bioinformatics, etc. This course introduces the fundamental statistical, mathematical, and computational concepts in analyzing data. The goal for this introductory course is to provide a solid foundation in the mathematics of machine learning, in preparation for more advanced machine learning concepts. The course focuses on univariate models, to simplify some of the mathematics and emphasize some of the underlying concepts in machine learning, including how should one think about data; how can data be summarized; how models can be estimated from data; what sound estimation principles look like; how generalization is achieved; and how to evaluate the performance of learned models.

Overview

• Basic probability concepts, covering both discrete and continuous cases
• Basic optimization concepts, needed for estimating models
• Analyzing scalar data
• Analyzing paired data
• Assessing model (prediction) performance

Learning Outcomes

By the end of the course, you should understand…

• The design process for solving a data analysis problem:
  • properties of the data
  • choosing a model
  • defining a computational problem (e.g. an optimization problem)
• Basic estimation algorithms, including maximum likelihood and linear regression and different optimization approaches for those problems
• Generalization, including the concept of over-fitting
• Evaluation of learned models

By the end of the course, you will have improved your skills in…

• Implementing basic estimation approaches (e.g., stochastic gradient descent for linear regression) in Julia
• Applying concepts from calculus and probability to solve real data problems
• Problem solving, by facing open-ended data analysis problems and needing to both formulate the problem and identifying appropriate algorithms to solve the problem

Topics

• Univariate probability
  • discrete distribution
  • continuous distribution
  • random variable
  • cdf, pdf
  • expectation
    • linearity
    • conditional expectation
  • variance
• Concentration phenomenon
  • basic central limit theorem
  • simple inequalities (Chebyshev, Hoeffding-Chernoff)
  • a foundation to understand generalization
• Estimation principles
  • maximum likelihood
  • maximum a posteriori
  • Bayesian approaches
  • regularization
• Optimizing functions
  • The role of derivatives and stationary points
  • First order and second order gradient descent
  • Batch and stochastic gradient descent
  • Step-size selection
• Regression
  • paired data
  • linear regression
  • polynomial regression
  • Bayesian regression
• Classification
  • logistic regression
• Regularization
  • The importance of constraining the function class
  • The impact on generalization error
• Measuring model performance
  • The role of a test set
  • Confidence intervals and statistical significance tests
  • Comparing two learned models

Knowledge Prerequisites

In this course, we will cover some basics in probability and optimization that you will need for the course. However, you will be applying these concepts for machine learning, and so it is important that (a) you have been exposed to some of the concepts before, and (b) are at least enforcing some of the mathematical knowledge in parallel. You must have taken calculus before this course, and have some programming experience. Background in probability and a first course in programming is recommended to be taken before this course, but can be taken as a co-requisite. An excitement to understand the mathematics underlying machine learning is a must.

The course CMPUT 272 is included as a co-requisite, as that course helps you become more comfortable with mathematical formalization. This co-requisite is particularly pertinent to those in CS, where CMPUT 272 is a requirement. For other departments, other math classes might provide that background, and can be used in place of CMPUT 272. In this case, you can email the instructor for permission to take the course without 272.

Pre-requisites

• One of MATH 100, 114, 117, 134 or 146
• CMPUT 174 or 274

Co-requisites

• One of STAT 141, 151, 235 or 265 or SCI 151
• MATH 125/127 (Linear algebra)
• CMPUT 272

Office hours/Labs

The instructor and TAs will hold frequent office hours. Occasionally, there may be lab sessions similar to office hours where students can ask TAs questions and work in the same room on the assignments. We may host review seminars, particularly early in the course, to cover background on Julia and background mathematical concepts.

Readings / Notes

Students are expected to read the corresponding sections about a class’s topic from notes before class as each class will discuss each topic in more detail and address questions about the material.

All readings are from the machine learning notes. These are designed to be short, so that you can read every chapter. I recommend avoiding printing these notes, since later parts of the notes are likely to be modified (even if only a little bit).

Other resources you can consult

• Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani, An Introduction to Statistical Machine Learning.
• T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning.
• David Barber, Bayesian Reasoning and Machine Learning.
• C.M. Bishop, Pattern Recognition and Machine Learning.
• Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong, Mathematics for Machine Learning.

Resources for Julia

• Getting Started with Julia

Grading

• Participation and Reading Questions : 10%
• Assignments (8, only best 7 count) : 30%
• Midterm exams (2, 15% each): 30%
• Final exam: 30%

There will be 8 assginments during the term and only the best 7 out of 8 will count towards your final grade. We will have weekly Participation and Reading Questions - mini quizzes, to be completed online. There will be 2 midterm exams. Marks will be converted to Letter Grades at the end of the course.

Late Policy

Participation and reading questions are given a 36 hour window and must be completed in that time. No late submissions will be accepted.

We will not accept late assignments; there is no late penalty policy. The assignments must be submitted electronically via eClass on time, by 11:59 pm Mountain time on the due date. There is a grace period of 48 hours when assignments will be accepted. No submissions will be accepted after 48 hours after the deadline, and the assignment will be considered as incomplete if not submitted. Refer to the FAQ document linked above for more information. If you have a personal issue (e.g., serious illness) that impacts your ability to submit on time, then please email the instructor asap before the deadline, so that an appropriate plan can be made.

Academic Honesty

All assignments and exams are individual, except when collaboration is explicitly allowed. All the sources used for problem solution must be acknowledged, e.g. web sites, books, research papers, personal communication with people, etc. Academic honesty is taken seriously; for detailed information see https://www.deanofstudents.ualberta.ca/en/AcademicIntegrity.aspx