D I S T R I B U T E D     O P T I M I Z A T I O N     A N D    G A M E S



Home     Publications

Distributed Optimization and Games, 2018-2019

The purpose of this course is to introduce students to large scale machine learning. The focus is as much on optimization algorithms as on distributed systems.

Lessons

You can freely use the slides below for your presentations, but I would like to be informed and please acknowledge the source in your presentation. Any comment is welcome. All lessons will be in Templiers room O+309 from 9.00 to 12.15.

Individual project

The individual project is an opportunity for the student to actively use the material taught in the course. The student is free to choose the goal of its project, but is invited to discuss it with the teacher. Possible goals are The mark will take into account: originality of the project, presentation quality, technical correctness, and task difficulty. Any form of plagiarism will lead to reduction of the final mark. A list of possible projects is provided below.

Submission rules

Ideas for possible projects

In random order, the list is extended during the course.
  1. Compare stochastic gradient descent and gradient aggregation methods in different regimes of condition-number versus dataset-size. See Bottou et al, section 5.3.3.
  2. Survey and compare iterage averaging methods. See Bottou et al, section 5.4.
  3. Survey the dynamic sampling techniques described in references [28,73] of Bottou et al (section 5.2.1). Implement and test at least one of them (including the basic one described in section 5.2.1).
  4. Implement the (inexact) Newton method without inverting the Hessian, but using the conjugate gradient method. Evaluate the effect of stopping the conjugate method after i ≤ d steps, where d is the dimension of the parameter vector. See Bottou et al section 6.1 and Bubeck section 2.4.
  5. Second-order techniques for neural network training. See Bottou et al section 6.1.2 and references [12,100] as a starting point. Better avoid this project if you are not familiar with backpropagation method for neural networks.
  6. Back to the classics: understand backpropagation from the original sources. Start from [134,135] in Bottou et al.
  7. Survey, implement and test L-BFGS. See Bottou et al section 6.2 and references [97,113].
  8. Asynchronicity as a momentum: ''Asynchrony begets Momentum, with an Application to Deep Learning'' by Mitliagkas et al, arxiv.
  9. ''Deep learning with Elastic Averaging SGD'' by Zhang et al, arxiv.

Exam


Last modified: January 29, 2019