For Data Science students: 25% classwork (a 10-minute test at every lesson, only 4 best marks will be considered), 50% theoretical exam, 25% lab evaluation.
Lessons
Lessons will be from 9 to 12 on Tuesdays in room 281, Lucioles Campus (1645 route des Lucioles, 06410 BIOT).
First lesson (Giovanni Neglia, January 16): introduction to the course,
introduction to FL (First papers [mcmahan16], [konecny16], in production, FL as a service, TensorFlow Federated, Motivations for FL, Cross-device and cross-silo scenario, Research directions: Efficiency, Fairness, Personalization, Privacy, Robustness to attacks and failures, Surveys [kairouz21fl_survey], [li20fl_survey]),
Some general background on ML (Minimize the expected loss, Working with the empirical loss, Generalisation bounds (VC-dim, Rademacher complexity, stability,...), Concrete algorithms, The workhorse: stochastic gradient method and its theoretical guarantees, Why not more sophisticated approaches?, Momentum and batches).
Second lesson (Giovanni Neglia, January 23): basic algorithms for federated learning, clients' full participation (Which loss should we select? Generalization bounds, from a centralized algorithm to a parallel one, sending gradients or sending models, multiple local steps [mishchenko22], a practical algorithm for cross-silo FL).
First Lab (Angelo Rodio, January 30).
Third lesson (Giovanni Neglia, February 6): clients' partial participation (Sampling may introduce a bias), case of independent client participations: how to remove the bias, the effect of time correlation [rodio22], learning rate tuning,
case of dependent client participation: FedAvg [mcmahan16], the two easy sampling cases [li20], difficulty to compute the actual bias (combinatorial problem) [ribeiro22], estimations problems in large populations,
communication cost (reduction?), compression, in-network aggregation, theoretical work from Peter Richtarik, more practical one from Marco Canini, Sampling.
Fourth lesson (Giovanni Neglia, February 13): Decentralized Federated Learning, Topology design for cross-silo FL: Role of the spectral gap [neglia20aistats];
Personalized Federated Learning, Naive approach [jian19], Model-Agnostic Meta-Learning [fallah20], Clustered FL [sattler20], Interpolation [mansour20], Learning a shared representation [collins21, marfoq23icml].
Second Lab (Angelo Rodio, February 20).
Fifth lesson (Chuan Xu, March 5). Threat models for FL, gradient inversion attack [Geiping2020], source inference attack [Hu2021], local model inversion attack [Xu2021], data poisoning attack [Tolpegin2020], model inversion attack [fredriksonModelInversionAttacks2015], differentially private SGD [McMahan2016], Byzantine resilient algorithm [Blanchard2017], secure aggregation [bonawitzPracticalSecureAggregation2017], TEE.
Third Lab and exam (Chuan Xu, March 12).
Labs
During these practical sessions students will have the opportunity to train ML models in a distributed way on Inria scientific cluster.
Sessions are organized by Angelo Rodio and Chuan Xu.
Students need to carry out some administrative/configuration steps before the start of the labs. Labs website.
Exam
The exam will be on March 12th between 9.00 and 12.00.
References
[li20fl_survey]
T. Li, A. K. Sahu, A. Talwalkar and V. Smith, "Federated Learning: Challenges, Methods, and Future Directions," in _IEEE Signal Processing Magazine_, vol. 37, no. 3, pp. 50-60, May 2020, doi: 10.1109/MSP.2020.2975749.
[kairouz21fl_survey]
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konecný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Hang Qi, Daniel Ramage, Ramesh Raskar, Mariana Raykova, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, Sen Zhao, Advances and Open Problems in Federated Learning, 2021
[mcmahan16]
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera y Arcas, Communication-Efficient Learning of Deep Networks from Decentralized Data, AISTATS 2017 (Arxiv version in February 2016)
[mishchenko22]
Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, Peter Richtárik, ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!, ICML 2022
[konecny16]
Konecný, J., McMahan, H. B., Ramage, D., & Richtárik, P. (2016). Federated Optimization: Distributed Machine Learning for On-Device Intelligence. _CoRR_, _abs/1610.02527_
[wright22]
S. Wright, B. Recht, Optimization for Data Analysis, Cambridge University Press, 2022
[marfoq22]
O. Marfoq, G. Neglia, L. Kameni, and R. Vidal, Federated Learning for Data Streams, AISTATS, 2023
[rodio22]
A. Rodio, O. Marfoq, F. Faticanti, G. Neglia, E. Leonardi, under submission
[li20]
X. Li, K. Huang, W. Yang, S. Wang, Z. Zhang, On the Convergence of FedAVG on Non-IID Data, ICLR, 2020
[stich19]
Sebastian U Stich, Local SGD Converges Fast and Communicates Little, ICLR, 2019
[woodworth20]
Blake Woodworth, Kumar Kshitij Patel, Sebastian U Stich, Zhen Dai, Brian Bullins, H Brendan McMahan, Ohad Shamir, Nathan Srebro, Is Local SGD Better than Minibatch SGD?, ICML, 2020
[ribeiro22]
M. Ribero, H. Vikalo, and G. De Veciana, “Federated Learning Under Intermittent Client Availability and Time-Varying Communication
Constraints,” arXiv preprint arXiv:2205.06730, 2022
[neglia20aistats]
Giovanni Neglia, Chuan Xu, Don Towsley, and Gianmarco Calbi, Decentralized gradient methods: does topology matter?. 23rd International Conference on Artificial Intelligence and Statistics (AISTATS). Palermo, Italy, June 2020
[neglia19infocom]
G. Neglia, G. Calbi, D. Towsley, G. Vardoyan, The Role of Network Topology for Distributed Machine Learning, Proc. of the IEEE International Conference on Computer Communications (INFOCOM 2019), Paris, France, April 29 - May 2, 2019
[marfoq20neurips]
Othmane Marfoq, Chuan Xu, Giovanni Neglia, and Richard Vidal,
Throughput-Optimal Topology Design for Cross-Silo Federated Learning, Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS 2020), December 6-12, online conference
[lebars22]
B Le Bars, A Bellet, M Tommasi, E Lavoie, AM Kermarrec, Refined Convergence and Topology Learning for Decentralized Optimization with Heterogeneous Data, arXiv preprint arXiv:2204.04452
[vogels22]
T Vogels, H Hendrikx, M Jaggi, Beyond spectral gap: The role of the topology in decentralized learning, NeurIPS 2022 - Advances in Neural Information Processing Systems 1 2022
[bishopPatternRecognitionMachine2016]
Christopher M. Bishop, Pattern Recognition and Machine Learning, 2016
[bonawitzPracticalSecureAggregation2017]
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, Karn Seth, Practical Secure Aggregation for Privacy-Preserving Machine Learning, 2017
[hubaPAPAYAPracticalPrivate2022]
Dzmitry Huba, John Nguyen, Kshitiz Malik, Ruiyu Zhu, Mike Rabbat, Ashkan Yousefpour, Carole-Jean Wu, Hongyuan Zhan, Pavel Ustinov, Harish Srinivas, Kaikai Wang, Anthony Shoumikhin, Jesik Min, Mani Malek, PAPAYA: Practical, private, and scalable federated learning, 2022
[michaelrabbatFederatedLearningScale2022]
“Federated Learning at Scale” Prof. Mike Rabbat, Meta AI, https://www.youtube.com/watch?v=j5e7bZLTMH0
[yinSeeGradientsImage2021]
Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M. Alvarez, Jan Kautz, Pavlo Molchanov, See through Gradients: Image Batch Recovery via GradInversion, 2021
[fredriksonModelInversionAttacks2015]
Matt Fredrikson, Somesh Jha, Thomas Ristenpart, Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures, 2015
[fallah20]
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar, Personalized federated learning: A meta-learning approach, NeurIPS 2020
[jiang19]
Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan, Improving federated learning personalization via model agnostic meta learning, arXiv:1909.12488 (2019). Presented at NeurIPS FL workshop 2019
[sattler20]
Felix Sattler, Klaus-Robert Müller, and Wojciech Samek, Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints, IEEE Transactions on Neural Networks and Learning Systems, 2020
[marfoq22neurips]
Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal, Federated Multi-Task Learning under a Mixture of Distributions, NeurIPS, 2022
[marfoq23icml]
Othmane Marfoq, Giovanni Neglia, Laetitia Kameni, and Richard Vidal, Personalized Federated Learning through Local Memorization, ICML, 2023
[smith17]
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar, Federated Multi-Task Learning, NIPS, 2017
[mansour20]
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh, Three approaches for personalization with applications to federated learning, arXiv:2002.10619
[collins21]
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai, Exploiting shared representations for personalized federated learning, ICML 2021
[Geiping2020]
Inverting gradients–How easy is it to break privacy in federated learning? NeurIPS 2020
[Hu2021]
Source Inference Attacks in Federated Learning, ICDM 2021
[Xu2021]
Chuan Xu, Giovanni Neglia. What else is leaked when eavesdropping Federated Learning? CCS workshop Privacy Preserving Machine Learning (PPML), 2021
[Tolpegin2020]
Data Poisoning Attacks Against Federated Learning Systems, ESORICS 2020
[McMahan2016]
Deep Learning with Differential Privacy, CCS 2016
