Samir M. Perlaza | INRIA | Permanent Member of the Scientific Staff

The Role of Relative Entropy in Supervised Machine Learning

Dates
- July 28, 2022 Electrical and Computer Engineering Department and Center for Statistics and Machine learning at Princeton University, Princeton NJ USA. Host: Prof. H. Vincent Poor
- July 26, 2022 Electrical and Computer Engineering Department at Rensselaer Polytechnic Institute, Troy NY USA. Host: Prof. Ali Tajer.
- July 6, 2022 Laboratoire Traitement et Communication de l'Information (LTCI), TelecomParis, Saclay, France. Host: Prof. Michèle Wigger.
Abstract
In this talk, recent results on various aspects of the Empirical Risk Minimization (ERM) problem with Relative Entropy Regularization (ERM-RER) are presented. The regularization is with respect to a sigma-finite measure, instead of a probability measure, which provides a larger flexibility for including prior knowledge on the models. Special cases of this general formulation include the ERM problem with (discrete or differential) entropy regularization and the information-risk minimization problem. Three results are discussed. First, it is shown that the empirical risk observed when models are sampled from the ERM-RER optimal probability measure is a sub-Gaussian random variable that exhibits a probably-approximately-correct (PAC) guarantee for the ERM problem. Second, the sensitivity of the expected empirical risk to deviations from the ERM-RER-optimal measure is characterized. Finally, using the notion of sensitivity, the impact of data aggregation on the generalization capabilities of machine learning algorithms based on the ERM-RER is studied. Interestingly, none of these results relies on statistical assumptions on the datasets, and thus, cases in which datasets exhibit different statistical properties can also be studied within this framework.
References
- Research Report, INRIA, No. RR-9454, Sophia Antipolis, France, Feb., 2022.
  ‘‘Empirical Risk Minimization with Generalized Relative Entropy Regularizations’’.
- Research Report, INRIA, No. RR-9474, Sophia Antipolis, France, Jun., 2022.
  ‘‘Sensitivity of the Gibbs Algorithm to Data Aggregation in Supervised Machine Learning’’.

I am interested in theoretical and practical aspects of game theory and information theory. The application domains that motivate my research include communication systems, power systems, behavioral influence, and distributed decision making.

Samir M. Perlaza | INRIA | Permanent Member of the Scientific Staff

Seminars, Lectures, and Talks

Research

Teaching

Academic Appointments

Education

Editorial Boards

International Research Projects

Acknowledgments and Awards