Statistical Machine Learning - Literature and suggested reading

The course literature in this course will be a textbook, where some of the authors also serve as lectures in the course The textbook is available at smlbook.org.

The book is not yet published and we are therefore very happy to receive any feedback or mistakes you have spotted. Follow the link above to provide feedback.

Reading instructions

Lecture	Chapters/sections	Comments
1. Introduction	2.1
2. Linear regression, regularization	3.1, 5.2, 3.A
3. Classification, Logistic regression	3.2, 4.5
4. LDA, QDA, kNN	11.1, 2.2
5. Cross validation, Bias-variance trade-off	4.1-4.4
6. Tree-based methods, bagging, random forests	2.3,2.A, 7.1, 7.2
7. Boosting	7.3, 5.1, (7.4)
8-9. Deep learning	6, 5.3-5.4	You may skip dropout in 6.3

Recommended supplementary reading

There are by now many resources written on the machine learning subject and new books keeps appearing all the time. Here are links to a few additional resources:

Compact general books

• Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani. An Introduction to Statistical Learning with Applications in R (freely available), Springer, 2013. We used this book as course literature in this course previously, and it follows the course relatively closely.
• Andriy Burkov. The hundred-page machine learning book (freely available), 2019. A good and well-written overview of modern machine learning. Not as technically detailed as many other books, but still very accurate.
• Yaser S. Abu-Mostafa, Malik Magdon-Ismalil, Hsuan-Tien Lin. Learning from data, AMLbook.com, 2012. A very well-written book with a focus on a thorough understanding of the generalization error and the bias-variance trade-off (at the cost of advanced methods, such as boosting and deep learning, which are not covered). This book was our main inspiration when we wrote chapter 4 in the SML book.

Extensive general books

• Trevor Hastie, Robert Tibshirani and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference and Prediction (freely available), Second edition, Springer, 2009. This book gives a more mathematical treatment of the subject. Particularly the chapters on linear regression and ridge regression, LDA and QDA are a good complement for the mathematically inclined student.
• Kevin P. Murphy. Machine learning - a probabilistic perspective (available as e-book via the library), MIT Press, 2012. With a scope extending well beyond this course, this books gives an introduction to many interesting subjects within machine learning.
• Christopher M. Bishop. Pattern Recognition and Machine Learning (freely available), Springer, 2006. Another general book on machine learning. This book was used as course literature for our PhD level course on machine learning.
• David Barber. Bayesian Reasoning and Machine Learning (freely available), Cambridge University Press, 2017. This book has a Bayesian perspective, meaning that probability theory is used heavily to reason about models, unknown parameters etc. The content of this book is not really in the scope of this course, but nevertheless interesting and also currently subject to a lot of research.

Other good books

• Marc Peter Deisenroth, A Aldo Faisal, and Cheng Soon Ong. Mathematics for Machine Learning (draft freely available), to be published by Cambridge University Press. A very recent book with focus on the mathematical tools for modern machine learning.
• Bradley Efron and Trevor Hastie. Computer Age Statistical Inference: Algorithms, Evidence and Data Science (freely available), Cambridge University Press, 2016. This books gives an historical perspective on the development of statistical methods since the arrival of the computer. With its statistical focus, this book is best suited for the mathematically inclined student.

Boosting

• Raúl Rojas. AdaBoost and the Super Bowl of Classifiers-A Tutorial Introduction to Adaptive Boosting. An introduction to boosting and AdaBoost, an alternative to 7.3 in the SML book.

Deep learning

• Yann LeCun, Yoshua Bengio & Geoffrey Hinton. Deep learning (access via the university network), Nature 521, 2015. A well-written introductory paper published in Nature, as a complement to chapter 6 in the SML book.
• Ian Goodfellow, Yoshua Bengio & Aaron Courville. Deep learning (freely available), MIT Press, 2016. A textbook on deep learning, with an introduction as well as state-of-the-art, obviously covering much more than this course.
• Daniel Geng & Rishi Veerapaneni. Tricking Neural Networks: Create your own Adversarial Examples, blog post 2018. A friendly introduction to adversarial examples, including (python) code for generating your own adversarial examples for the same data set (MNIST) as we use in the lab.