Markov chains and their control, general theory, constrained MDPs

This area of research goes back to my Ph.D. thesis (under the guidance of Prof. Adam Shwartz), and I have always remained faithful to it. My main interest is in constrained Markov decision processes, for which the standard dynamic programming techniques are not applicable; the computation of optimal policies requires linear programming or Lagrange techniques. I have been working on different aspects, such as adaptive control, sensitivity analysis, finite-state approximations.

Monographs:

E. Altman , Constrained Markov Decision Processes, Chapman and Hall/CRC , 1999.

For the postscript version of the contents, introduction, references and index, click here. To get an ascii version of the contents click here.

Papers:

E. Altman and A. Shwartz , Markov decision problems and state-action frequencies , SIAM J. Control and Optimization , Vol. 29, No. 4, pp. 786-809, 1991.
E. Altman and A. Shwartz , Adaptive Control of constrained Markov chains , IEEE Trans. Auto. Control , Vol. 36 No. 4, pp. 454-462, 1991.
E. Altman and A. Shwartz , Adaptive Control of constrained Markov chains: Criteria and Policies:w, Annals of Operations Research , Vol. 28, special issue on ``Markov Decision Processes", Eds. O. Hernandez-Lerma and J. B. Lasserre, pp. 101-134, 1991.
E. Altman and A. Shwartz , Sensitivity of constrained Markov Decision Problems , Annals of Operations Research , Vol. 32, pp. 1-22, 1991.
E. Altman and A. Shwartz , Time-sharing policies for controlled Markov chains , Operations Research , Vol. 41, No. 6, pp. 1116-1124, 1993.
E. Altman and V. A. Gaitsgory , Stability and Singular Perturbations in Constrained Markov Decision Problems , IEEE Trans. Auto. Control , Vol. 38, No. 6, pp. 971-975, 1993.
E. Altman , Asymptotic Properties of Constrained Markov Decision Processes , ZOR - Methods and Models in Operations Research , Vol. 37, Issue 2, pp. 151-170, 1993.
E. Altman , Denumerable Constrained Markov Decision Problems and Finite Approximations , Math. of Operations Research , Vol. 19, No. 1, pp. 169-191, 1994.
E. Altman and O. Zeitouni , Rate of convergence of empirical measures and costs in controlled Markov chains and transient optimality , Math. of Operations Research , Vol. 19, No. 4, pp. 955-974, 1994.
M. Tidball and A. Altman , Continuity of optimal values and solutions of convex optimization, and constrained control of Markov chains , submitted to SIAM , 1994.
E. Altman and F. Spieksma , The Linear Program approach in Markov Decision Problems revisited [ps, pdf ] ZOR - Methods and Models in Operations Research , Vol. 42, Issue 2, pp. 169-188, 1995.
E. Altman , Constrained Markov decision processes with total cost criteria: occupation measures and primal LP , ZOR - Methods and Models in Operations Research , Vol. 43, Issue 1, 1996.
E. Altman , Constrained Markov decision processes with total cost criteria: Lagrange approach and dual LP , ZOR - Methods and Models in Operations Research, Vol. 48, pp. 387-417, 1998.
E. Altman and G. Koole , On submodular value functions and complex dynamic programming , INRIA report No. 2658, 1995. Final version in Stochastic Models, pp. 1051-1072, 1998.
E. Altman, A. Hordijk and L. C. M. Kallenberg , On the value in constrained control of Markov chains , ZOR, Vol. 44, Issue 3, pp. 387-400, 1996.
E. Altman, K. E. Avrachenkov and J. A. Filar , Asymptotic linear programming and policy improvement for singularly perturbed Markov decision processes , ZOR - Methods and Models in Operations Research, , Vol. 49, pp. 97-110, 1999.
J. Filar, E. Altman, K. E. Avrachenkov , An asymptotic simplex method for singularly perturbed linear programs, , Operations Research Letters , v.30, no.5, pp.295-307, October 2002.
M. Tidball, A. Lombardi, O. Pourtallier and E. Altman, , Continuity of optimal values and solutions for control of Markov chains with constraints , SIAM J. Control and Optimization , Vol. 38, No. 4, pp. 1204-1222, 2000.
E. Altman, K. E. Avrachenkov, R. Nunez-Queija, Perturbation analysis for denumerable Markov chains with application to queueing models, Advances in Applied Probability, Vol 36, Number 3, pp. 839-853, 2004.


Back to my Web page .