For Data Science students: 25% classwork (a 10-minute test at every lesson, only 4 best marks will be considered), 50% theoretical exam, 25% lab evaluation.
Lessons will be from 13.30 to 16.45.
First lesson (Giovanni Neglia, January 11, online): introduction to the course,
introduction to ML optimization (empirical risk vs expected risk, training/validation/test sets). Sections 1-3.1 of Bottou et al.
Second lesson (Giovanni Neglia, January 18, online):
math refresher (gradient, hessian, convex sets and functions),
presentation of full batch gradient and stochastic gradient methods,
why stochastic gradient descent (SGD) may outperform batch gradient (qualitative explanation, time to minimize the empirical error), overview of noise reduction and second order methods, mini-batch methods. Section 4.3 of Goodfellow et al, Sections 3.2, 3.3, introduction to Section 4 of Bottou et al.
Third lesson (Giovanni Neglia, January 25): convergence results (expected decrease after one iteration), definition of strongly convexity. Sections 4.1 and 4.2 of Bottou et al.
Fourth lesson (Giovanni Neglia, February 1): convergence results of stochastic gradient methods for strongly convex functions (both constant and decreasing learning rates), the role of the condition number. Section 4.2 of Bottou et al.
Fifth lesson (Othmane Marfoq, February 8): convergence results of stochastic gradient methods for non-convex functions (with both constant and decreasing learning rates), noise reduction methods (dynamic sample size, gradient aggregation). Section 4.3, 5.1, 5.2, and 5.3 of Bottou et al.
Sixth lesson (Othmane Marfoq, February 15): other optimization methods (momentum, Nesterov, coordinate descent method), second order methods (Newton, Hessian-free inexact, quasi-Newton, Gauss-Newton methods), introduction to neural networks. Sections 6.1, 6.2, and 6.3 of Bottou et al.
During these practical sessions students will have the opportunity to train ML models in a distributed way on Inria scientific cluster.
Sessions are organized by Othmane Marfoq and Angelo Rodio.
Students need to carry out some administrative/configuration steps before the start of the labs. Labs website.
Participation to the labs will be graded by the teachers.
The exam will be on February 22nd between 13.30 and 16.30.