R.R.3. Theoretical Reading List C (Probability Theory)
Some of the section of the Learning Theory Coursebook requires some papers and some books and materials, mostly through and through probability theory, statistical learning theory, and statistics (mathematical statistics in general), alongside some materials about concentration inequalities.
Probability and Statistics
We have, for now, some resources being the foundational books and materials as followed. First, for probability theory:
- Basic Probability Theory (Sharon Goldwater, University of Edinburgh) - here.
- An Introduction to Probability Theory (Cristin Toninelli) - here.
- Wiley Series in Probability and Mathematical Statistics - An introduction to Probability Theory and Its Applications (William Feller) - here (volume I).
- Probability Theory - The Logic of Science (E. T. Jaynes) - here.
- Probability: Theory and Examples (Rick Durrett) - here.
- Probability and Stochastic Processes with Applications (Oliver Knill) - here.
- Fundamental of Probability Theory (M. E. Harr, Purdue University) - here.
For statistics:
Statistical Learning Theory
For statistical learning theory, major works are enough often time. For example, we will have canonical works like (Russell and Norvig 2009) and (Mohri et al. 2012). However, there are, for example:
- Understanding Machine Learning: From Theory to Algorithms as in (Shalev-Shwartz and Ben-David 2014).
- Introduction to Statistical Learning Theory (Oliver Bousquet, Stephane Boucheron, Gabor Lugosi) - here.
- Statistical Learning Theory: Models, Concepts, and Results (Ulrike von Luxburg, Bernhard Schoelkopf) - here.
- An Introduction to Statistical Learning (with Application in R) - here.
- ECE 543: Statistical Learning Theory (Bruce Hajek and Maxim Raginsky) - (Hajek and Raginsky 2021) and can be found here.
- A Theory of Universal Learning (Bousquet et al.) - here.
- An Overview of Statistical Learning Theory (V. N. Vapnik) - here.
- Vladimir N. Vapnik’s The Nature of Statistical Learning - (Vapnik 1999).
- Math 547. Statistical Learning Theory, Fall 2019 (Steven Heilman) - Here.
- Statistical learning in high dimension: a rigorous statistical physics approach (Doctorate Thesis, Cedric Gerbelot) - Here.
- CPSC 532D — Statistical Learning Theory (Danica J. Sutherland, University of British Columbia, Vancouver) - Here.
- An Elementary Introduction to Statistical Learning Theory (Sanjeev Kulkarni, Gilbert Harman) - This can be taken from here inside this repository.
Those are the majority of sources that would be presented mostly coherent, by some and obvious standards. Some of them are lecture notes, and while some like the overview of SLT is a paper, most are standard references in the field, which I believe would be rather good. That said, we do have something like:
- While (Mohri et al. 2012) and (Shalev-Shwartz and Ben-David 2014) is standard, both of them are essentially the same with only added semantic on the notation part. At that, not a lot even happened.
- Many sources are limited to simple binary classification then jump straight into general case. Most probable, their learning setting is extremely limited.
Some observations at that can be taken into account, and also the fact about the system in general. In the same length, Hastie’s The Elements of Statistical Learning (which can be accessed here), is rather trivial in hindsight.