Control of Markov chains: applications

We have been working mainly on applications of MDPs in queueing systems. We also have a survey on applications of MDPs to communication networks ( postscript file , pdf file ), that appeared in Handbook of Markov Decision Processes Methods and Applications, edited by E. A. Feinberg and A. Shwartz, Kluwer, 2001. Quite often dynamic programming techniques can be used in order to obtain the structure of optimal policies in the control of queueing systems, even when the state space is infinite. This then allows to reduce a dynamic control problem into an optimization problem over a relatively small parameter set, and, in some cases, to fully solve the problem. Our work that concerns applications of MDPs to queueing network is mostly focused on establishing the structure of optimal policies and values.

Papers:

E. Altman and A. Shwartz , Optimal priority assignment: a time sharing approach , IEEE - Transactions on Automatic Control , Vol. 34, No. 10, pp. 1098-1102, 1989.
E. Altman and P. Nain , Optimal Control of an M/G/1 Queue with Vacations , IEEE - Transactions on Automatic Control , Vol. 38, No. 12, pp. 1766-1775, 1993.
E. Altman and P. Nain , Optimality of a threshold policy in the M/M/1 queue with repeated vacations , ZOR - Mathematical Methods of Operations Research , Vol. 44, Issue 2, pp. 75-96, 1996.
E. Altman and H. J. Kushner , Admission Control for Combined Guaranteed Performance and Best Effort Communications Systems Under Heavy Traffic , SIAM J. Control and Optimization , pp. 1780-1807, vol 37 no 6, 1999.
E. Altman and H. J. Kushner , Control of Polling in Presence of Vacations in Heavy Traffic with Applications to Satellite and Mobile Radio Systems , to appear in SIAM J. Control and Optimization , 2001.
E. Altman, G. Koole and T. Jimenez , On optimal call admission control in a resource-sharing system , IEEE Trans. on Communications , Volume 49, Number 9, pp. 1659-1668, 2001.
Eitan Altman , Applications of Markov Decision Processes in Communication Networks: a Survey ( pdf , ps ), in Handbook of Markov Decision Processes Methods and Applications, edited by E. A. Feinberg and A. Shwartz, Kluwer, 2001.
W. K. Ching, M. K. Ng, K. K. Wong and E. Altman, , Customer lifetime value: stochastic optimization approach , Journal of the Operational Research Society , Vol 55, pp 860-868, 2004.

Control and performance evaluation of systems with delayed information ,
and in
Stochastic games , where several controllers are considered.

Back to my Web page .