Steven I. Marcus

1 TWO TIMESCALE SPSA ALGORITHMS FOR RATE-BASED ABR (2008)

We propose a randomized search method called Stochastic Model Reference Adaptive Search (SMRAS) for solving stochastic optimization problems in situations where the objective functions cannot be...

Flow Control, Shalabh Bhatnagar, Michael C. Fu, Steven I. Marcus

Jiaqiao Hu, Michael C. Fu, Vahid R. Ramezani, Steven I. Marcus

informs ®

Optimal Joint Preventive Maintenance and Production Policies* (2008)

Xiaodong Yao, Xiaolan Xie, Michael C. Fu, Steven I. Marcus

Abstract: We study joint preventive maintenance (PM) and production policies for an unreliable production-inventory system in which maintenance/repair times are non-negligible and stochastic. A joint...

Index Terms (2008)

Hyeong Soo Chang, Michael C. Fu, Jiaqiao Hu, Steven I. Marcus

We present a sampling algorithm, called “Recursive Automata Sampling Algorithm ” (RASA), for control of finite horizon Markov decision processes. By extending in a recursive manner the learning...

Scott B. Laprise, Michael C. Fu, Steven I. Marcus, Huiju Zhang

informs ® doi 10.1287/mnsc.1050.0447

Risk-Sensitive Optimal Control Of Hidden Markov Models: Structural Results (2008)

Emmanuel Fernández-Gaucherand, Steven I. Marcus

this paper results of an investigation on the nature and structure of risk-sensitive controllers. We pose the following question: How does risk-sensitivity manifest itself in the structure of a...

A Model Reference Adaptive Search Method for Global Optimization (2008)

A Model Reference Adaptive Search Method for Global Optimization (2008)

Modeling Stochastic Discrete Event Systems Using Probabilistic Languages (2007)

We introduce a new randomized method called Model Reference Adaptive Search (MRAS) for solving global optimization problems. The method works with a parameterized probabilistic model on the solution...

Vijay K. Garg, Ratnesh Kumar, Steven I. Marcus

The formalism of probabilistic languages has been introduced for modeling the qualitative behavior of stochastic discrete event systems. A probabilistic language is a unit interval valued map over...

Corrections to "Finite Buffer Realization of Input-Output Discrete Event Systems" (2007)

Extension based Limited Lookahead Supervision of Discrete Event Systems (2007)

This note presents a correction to [1, Theorem 4] which provides a necessary and sufficient condition for dispatchability. 1 Notation The following additional notation is introduced: The notation s...

Ratnesh Kumar, Hok M. Cheung, Steven I. Marcus

Supervisory control of discrete event systems using limited lookahead has been studied by Chung-Lafortune-Lin, where control is computed by truncating the plant behavior up to the limited lookahead...

A Framework for Mixed Estimation of Hidden Markov Models (2007)

Subhrakanti Dey, Steven I. Marcus

In this paper, we present a framework for a mixed estimation scheme for hidden Markov models (HMM). A robust estimation scheme is first presented using the minimax method that minimizes a worst case...

Rate Based ABR Flow Control using Two Timescale SPSA (2007)

A Framework for Mixed Estimation of Hidden Markov Models (2007)

In this paper, a two timescale simultaneous perturbation stochastic approximation (SPSA) algorithm is developed and applied to closed loop rate based available bit rate (ABR) flow control. The...

Subhrakanti Dey, Steven I. Marcus

In this paper, we present a framework for a mixed estimation scheme for hidden Markov models (HMM). A robust estimation scheme is first presented using the minimax method that minimizes a worst case...

Risk-Sensitive Queueing (2007)

Risk-Sensitive, Minimax, and Mixed Risk-Neutral/Minimax Control of Markov Decision Processes (2007)

This paper has two objectives. The first is to introduce a unifying framework for risk-sensitive control of Markov Decision Processes (MDPs). The framework includes, as special cases, a number of...

Stefano Coraluppi Steven, Steven I. Marcus

: This paper analyzes a connection between risk-sensitive and minimax criteria for discrete-time, finite-state Markov Decision Processes (MDPs). We synthesize optimal policies with respect to both...

Observations on Nonlinear Risk-Sensitive Control (2007)

Risk-Sensitive, Minimax, and Mixed Risk-Neutral/Minimax Control of Markov Decision Processes (2007)

This paper clarifies the relationship between risksensitive and robust control. This topic has received attention in the recent literature. We show that the formulations are related through an...

Controlled Switching Diffusions as Hybrid Processes (2007)

: This paper analyzes a connection between risk-sensitive and minimax criteria for discrete-time, finite-state Markov Decision Processes (MDPs). We synthesize optimal policies with respect to both...

Mrinal K. Ghosh, Steven I. Marcus, Aristotle Arapostathis

this paper we discuss controlled switching diffusions as a particular class of hybrid control systems. This class of systems is also of interest in the study of deterministic hybrid systems, since...

On The Average Cost Optimality Equation And The Structure Of Optimal Policies For Partially Observable Markov Decision Processes (2007)

Emmanuel Fernández-Gaucherand, Emmanuel Fern, Aristotle Arapostathis, Steven I. Marcus

We consider partially observable Markov decision processes with finite or countably infinite (core) state and observation spaces and finite action set. Following a standard approach, an equivalent...

Stochastic Differential Games With Multiple Modes (2007)

Mrinal K. Ghosh, Steven I. Marcus

We have studied two person stochastic differential games with multiple modes. For the zero-sum game we have established the existence of optimal strategies for both players. For the nonzero sum case...

Risk-Sensitive Optimal Control Of Hidden Markov Models: Structural Results (2007)

Emmanuel Fernández-Gaucherand, Steven I. Marcus

this paper results of an investigation on the nature and structure of risk-sensitive controllers for HMM. We pose the following question: How does risk-sensitivity manifest itself in the structure of...

Risk Sensitive Control of Markov Processes in Countable State Space (2007)

Daniel Hern'andez-Hern'andez, Steven I. Marcus

In this paper we consider infinite horizon risk-sensitive control of Markov processes with discrete time and denumerable state space. This problem is solved proving, under suitable conditions, that...

Session T2C A NEW PEDAGOGY IN ELECTRICAL AND COMPUTER ENGINEERING: AN EXPERIENTIAL AND CONCEPTUAL APPROACH (2007)

Zeynep Dilli, Neil Goldsman, Janet A. Schmidt, Lee Harper, Steven I. Marcus

Abstract ⎯ This paper reports on an experimental program for educating high-school and beginning-college students in ubiquitous areas within Electrical and Computer Engineering (ECE). The unique...

1 (2007)

Steven I. Marcus, Pedram Fard

Risk-sensitive control is an area of significant current interest in stochastic control theory. It is a generalization of the classical, risk-neutral approach, whereby we seek to minimize an...

Harvard University, and Industry Corrections to \Finite Bu er Realization of Input-Output Discrete Event Systems" (2007)

R. Kumar, V. K. Garg, S. I. Marcus, Ratnesh Kumar, Vijay K. Garg, Steven I. Marcus

This note presents a correction to [1, Theorem 4] which provides a necessary and su cient condition for dispatchability. 1 Notation The following additional notation is introduced: The notation s L s...

B.A.Peters,J.S.Smith,D.J.Medeiros,andM.W.Rohrer,eds. ABSTRACT A NEW APPROACH TO PRICING AMERICAN-STYLE DERIVATIVES (2007)

Scott B. Laprise, Steven I. Marcus

This paper presents a new approach to pricing Americanstyle derivatives. By approximating the value function with a piecewise linear interpolation function, the option holder’s continuation value...

An Evolutionary Random Policy Search Algorithm for Solving Markov Decision Processes (2007)

Jiaqiao Hu, Michael C. Fu, Vahid R. Ramezani, Steven I. Marcus

This paper presents a new randomized search method called evolutionary random policy search (ERPS) for solving infinite-horizon discounted-cost Markov-decision-process (MDP) problems. The algorithm...

An asymptotically efficient simulation-based algorithm for finite horizon stochastic dynamic programming (2007)

Hyeong Soo Chang, Michael C. Fu, Steven I. Marcus

We present a simulation-based algorithm called “Simulated Annealing Multiplicative Weights” (SAMW) for solving large finite-horizon stochastic dynamic programming problems. At each iteration of...

A Note on an LQG Regulator With Markovian Switching and Pathwise Average Cost (2006)

Ghosh, Mrinal K., Arapostathis, Aristotle, Marcus, Steven I.

We study a linear system with a Markovian switching parameter perturbed by white noise. The cost function is quadratic. Under certain conditions, we find a linear feedback control which is almost...

Analysis of an Adaptive Control Scheme for a Partially Observed Controlled Markov Chain (2006)

Fernandez-Gaucherand, Emmanuel, Arapostathis, Aristotle, Marcus, Steven I.

The authors consider an adaptive finite state-controlled Markov chain with partial state information, motivated by a class of replacement problems. They present parameter estimation techniques based...

Optimal Control of Switching Diffusions With Application to Flexible Manufacturing Systems (2006)

Ghosh, Mrinal K., Arapostathis, Aristotle, Marcus, Steven I.

A controlled switching diffusion model is developed to study the hierarchical control of flexible manufacturing systems. The existence of a homogeneous Markov non-randomized optimal policy is...

Discrete-Time Controlled Markov Processes With Average Cost Criterion: A Survey (2006)

Arapostathis, Aristotle, Borkar, Vivek S., Fernandez-Gaucherand, Emmanuel, Ghosh, Mrinal K., Marcus, Steven I.

This work is a survey of the average cost control problem for discrete time Markov processes. We have attempted to put together a comprehensive account of the considerable research on this problem...

Predicates and Predicate Transformers for Supervisory Control of Discrete Event Dynamical Systems (2006)

Kumar, Ratnesh, Garg, Vijay, Marcus, Steven I.

Most discrete event system models are based on defining the alphabet set or the set of events as a fundamental concept. This paper takes an alternative view of treating the state space as the...

Language Stability and Stabilizability of Discrete Event Dynamical Systems (2006)

Kumar, Ratnesh, Garg, Vijay, Marcus, Steven I.

This paper studies the stability and stabilizability of Discrete Event Dynamical Systems (DEDS's) modeled by state machines. We define stability and stabilizability in terms of the behavior of the...

Ergodic Control of Switching Diffusions (2006)

Ghosh, Mrinal K., Arapostathis, Aristotle, Marcus, Steven I.

We study the ergodic control problem of switching diffusions representing a typical hybrid system that arises in numerous applications such as fault tolerant control systems, flexible manufacturing...

An Evolutionary Random Search Algorithm for Solving Markov Decision Processes (2005)

Hu, Jiaqiao, Fu, Michael C., Ramezani, Vahid, Marcus, Steven I.

A Model Reference Adaptive Search Algorithm for Global Optimization (2005)

Hu, Jiaqiao, Fu, Michael C., Marcus, Steven I.

An Evolutionary Random Search Algorithm for Solving Markov Decision Processes (2005)

Hu, Jiaqiao, Fu, Michael C., Ramezani, Vahid, Marcus, Steven I.

A Model Reference Adaptive Search Algorithm for Global Optimization (2005)

Hu, Jiaqiao, Fu, Michael C., Marcus, Steven I.

Evolutionary policy iteration for solving Markov decision processes (2005)

Hyeong Soo Chang, Hong-gi Lee, Michaelc Fu, Steven I. Marcus

[12] J. Schumacher, “Finite-dimensional regulators for a class of infinite-dimensionalsystems,”

An adaptive sampling algorithm for solving Markov decision processes (2005)

Hyeong Soo Chang, Michael C. Fu, Steven I. Marcus, Hyeong Soo Chang, Michael C. Fu, Steven I. Marcus

Based on recent results for multi-armed bandit problems, we propose an adaptive sampling algorithm that approximates the optimal value of a finite horizon Markov decision process (MDP) with infinite...

An adaptive sampling algorithm for solving Markov decision processes (2005)

Hyeong Soo Chang, Michael C. Fu, Jiaqiao Hu, Steven I. Marcus

A Model Reference Adaptive Search Method for Stochastic Global Optimization (2005)

Optimal preventive maintenance scheduling in semiconductor manufacturing (2004)

We propose a new method called Stochastic Model Reference Adaptive Search (SMRAS) for finding a global optimal solution to a stochastic optimization problem in situations where the objective...

Xiaodong Yao, Emmanuel Fernández-gaucher, Michael C. Fu, Steven I. Marcus

Preventive Maintenance (PM) scheduling is a very challenging task in semiconductor manufacturing, due to the complexity of highly integrated fab tools and systems, the interdependence between PM...

Risk-sensitive probability for Markov chains � (2004)

Vahid Reza Ramezani, Steven I. Marcus

The probability distribution of a Markov chain is viewed as the information state of an additive optimization problem. This optimization problem is then generalized to a product form whose...

Two-Timescale Simultaneous Perturbation Stochastic Approximation Using Deterministic Perturbation Sequences (2003)

Bhatnagar, Shalabh, Fu, Michael C, Marcus, Steven I, Wang, I-Jeng

Simultaneous perturbation stochastic approximation (SPSA) algorithms have been found to be very effective for high-dimensional simulation optimization problems. The main idea is to estimate the...

Two-Timescale Simultaneous Perturbation Stochastic Approximation Using Deterministic Perturbation Sequences (2003)

Bhatnagar, Shalabh, Fu, Michael C, Marcus, Steven I, Wang, I-Jeng

Simultaneous perturbation stochastic approximation (SPSA) algorithms have been found to be very effective for high-dimensional simulation optimization problems. The main idea is to estimate the...

An Asymptotically Efficient Algorithm for Finite Horizon Stochastic Dynamic Programming Problems (2003)

Chang, Hyeong Soo, Fu, Michael C., Marcus, Steven I.

We present a novel algorithm, called ``Simulated Annealing Multiplicative Weights", for approximately solving large finite-horizon stochastic dynamic programming problems. The algorithm is...

An Asymptotically Efficient Algorithm for Finite Horizon Stochastic Dynamic Programming Problems (2003)

Chang, Hyeong Soo, Fu, Michael C., Marcus, Steven I.

We present a novel algorithm, called ``Simulated Annealing Multiplicative Weights", for approximately solving large finite-horizon stochastic dynamic programming problems. The algorithm is...

Approximate receding horizon approach for markov decision processes: average reward case (2003)

Hyeong Soo Chang, Steven I. Marcus

We consider an approximation scheme for solving Markov Decision Processes (MDPs) with countable state space, finite action space, and bounded rewards that uses an approximate solution of a fixed...

Two-Timescale simultaneous perturbation stochastic approximation using deterministic perturbation sequences (2003)

Ying He, Michael C. Fu, Steven I. Marcus

In this paper, we consider Simultaneous Perturbation Stochastic Approximation (SPSA) for function minimization. The standard assumption for convergence is that the function be three times...

Convergence of simultaneous perturbation stochastic approximation for nondifferentiable optimization (2003)

Convergence of simultaneous perturbation stochastic approximation for nondifferentiable optimization (2003)

Simultaneous perturbation stochastic approximation (SPSA) algorithms have been found to be very effective for high-dimensional simulation optimization problems. The main idea is to estimate the...

Convergence of Simultaneous Perturbation Stochastic Approximation for Nondifferentiable Optimization (2003)

Simultaneous perturbation stochastic approximation (SPSA) algorithms have been found to be very effective for high-dimensional simulation optimization problems. The main idea is to estimate the...

Ying He, Michael C. Fu, Steven I. Marcus

Abstract—In this note, we consider simultaneous perturbation stochastic approximation for function minimization. The standard assumption for convergence is that the function be three times...

Algebraic Structure and Finite Dimensional Nonlinear Estimation, (2002)

Algebraic Structure and Finite Dimensional Nonlinear Estimation. (2002)

Optimal recursive state estimators have been derived for very general classes of nonlinear stochastic systems. The optimal estimator requires, in general, an infinite dimensional computation to...

The Use of Harmonic Analysis in Suboptimal Estimator Design. (2002)

The algebraic structure of certain classes of nonlinear systems is exploited in order to prove that the optimal estimators for these systems are recursive and finite-dimensional. These systems are...

The Use of Spherical Harmonics in Suboptimal Estimator Design, (2002)

The state extimation problem for bilinear stochastic systems evolving on compact Lie groups and homogeneous spaces is considered. The problem is motivated by some applications involving rotational...

The Use of Harmonic Analysis in Suboptimal Estimator Design. (2002)

The state estimation problem for bilinear stochastic systems evolving on sphere is considered. The problem is motivated by some applications involving rotational processes in three dimensions. Then...

Marcus,Steven I., Willsky,Alan S., Hsu,Kai

The state estimation problem for bilinear stochastic systems evolving on compact Lie groups and homogeneous spaces is considered. The problem is motivated by some applications involving rotational...

Stability Results for a Class of Systems with Multiplicative State Noise. (2002)

Wise,Gary L., Marcus,Steven I.

Presented at the Annual Allerton Conference on Communication, Control, and Computing(15th), 28-30 Sep 77.

Finite Dimensional Nonlinear Estimation in Continuous and Discrete Time. (2002)

Marcus,Steven I., Mitter,Sanjoy K., Ocone,Daniel

It has been shown that, for certain classes of nonlinear stochastic systems in both continuous and discrete time, the optimal conditional mean estimator of the system state given the past...

Estimator Performance for a Class of Nonlinear Estimation Problems. (2002)

Liu,Chang-Huan, Marcus,Steven I.

The state estimation problem for a certain class of nonlinear stochastic systems with white Gaussian plant and observation noise is considered. The optimal (minimum variance) estimators for these...

A General Martingale Approach to Discrete Time Stochastic Control and Estimation. (2002)

Hsu,Kai, Marcus,Steven I.

A general method of constructing system models for the solution of discrete time stochastic control and estimation problems is presented. The method involves the application of modern martingale...

Annual Report on Electronics Research at The University of Texas at Austin. (2002)

Powers,Edward J., Marcus,Steven I.

This report summarizes progress on projects carried out at the Electronics Research Center at The University of Texas at Austin and which were supported by the Joint Services Electronics Program. In...

Annual Report on Electronics Research at The University of Texas at Austin. (2002)

Powers,Edward J., Marcus,Steven I.

This report summarizes progress on projects carried out at the Electronics Research Center at The University of Texas at Austin and which were supported by the Joint Services Electronics Program. In...

The Lie Algebraic Structure of a Class of Finite Dimensional Nonlinear Filters. (2002)

Liu,Chang-Huan, Marcus,Steven I.

We present an example of the application of Lie algebraic techniques to nonlinear estimation problems. The method relates the computation of the (unnormalized) conditional density and the computation...

Approximate and Local Linearizability of Nonlinear Discrete-Time Systems, (2002)

Lee,Hong-Gi, Marcus,Steven I.

This document considers a single-input nonlinear discrete-time system of a certain form. Many authors have studied (local or global) linearization (Cheng et. al. 1985, Hunt and Su 1981, Jakubczyk and...

Multi-time Scale Markov Decision Processes (2002)

Chang, Hyeong Soo, Fard, Pedram, Marcus, Steven I., Shayman, Mark

This paper proposes a simple analytical model called M time-scale MarkovDecision Process (MMDP) for hierarchically structured sequential decision making processes, where decisions in each level in...

An Adaptive Sampling Algorithm for Solving Markov Decision Processes (2002)

Chang, Hyeong Soo, Fu, Michael C., Marcus, Steven I.

Based on recent results for multi-armed bandit problems, we propose an adaptive sampling algorithm that approximates the optimal value of a finite horizon Markov decision process (MDP) with infinite...

Risk-Sensitive Probability for Markov Chains (2002)

Ramezani, Vahid R., Marcus, Steven I.

The probability distribution of a Markov chain is viewed as the information state of an additive optimization problem. This optimization problem is then generalized to a product form whose...

Multi-time Scale Markov Decision Processes (2002)

Chang, Hyeong Soo, Fard, Pedram, Marcus, Steven I., Shayman, Mark

This paper proposes a simple analytical model called M time-scale MarkovDecision Process (MMDP) for hierarchically structured sequential decision making processes, where decisions in each level in...

An Adaptive Sampling Algorithm for Solving Markov Decision Processes (2002)

Chang, Hyeong Soo, Fu, Michael C., Marcus, Steven I.

Based on recent results for multi-armed bandit problems, we propose an adaptive sampling algorithm that approximates the optimal value of a finite horizon Markov decision process (MDP) with infinite...

Risk-Sensitive Probability for Markov Chains (2002)

Ramezani, Vahid R., Marcus, Steven I.

The probability distribution of a Markov chain is viewed as the information state of an additive optimization problem. This optimization problem is then generalized to a product form whose...

Seoul, Korea 156-756 (2002)

Hong-gi Lee, Ari Arapostathis, Senior Member, Steven I. Marcus

We extend the results in [14] to multi-input systems, and utilize these to obtain necessary and sufficient conditions for the linearization of discrete-time nonlinear systems via restricted dynamic...

Jointly Optimized Bit-Rate/Delay Control Policy for Wireless Packet Networks With Fading Channels (2002)

Javad Razavilar, Senior Member, Steven I. Marcus

Abstract—In this paper, we consider the downlink rate control problem in a wireless channel. A dynamic programming optimization method is introduced to obtain the optimal bit-rate/delay control...

Multi-time Scale Markov Decision Processes (2002)

Hyeong Soo Chang, Pedram Fard, Steven I. Marcus, Mark Shayman

This paper proposes a simple analytical model called time-scale Markov Decision Process (MMDP) for hierarchically structured sequential decision making processes, where decisions in each level in the...

A Model for Multi-time Scaled Sequential Decision Making Processes (2002)

Hyeong Soo Chang, Pedram Fard, Steven I. Marcus, Mark A. Shayman

We propose a simple analytical model called M time-scale Markov Decision Process (MMDP) for hierarchically structured sequential decision making processes, where decisions in each level in the...

Estimation of Hidden Markov Models (2001)

Ramezani, Vahid Reza, Marcus, Steven I.

A risk-sensitive generalization of the Maximum A Posterior Probability (MAP) estimationfor partially observed Markov chains is presented.Using a change of measure technique,a cascade filtering scheme...

Approximate Receding Horizon Approach for Markov Decision Processes: Average Reward Case (2001)

Markov Games: Receding Horizon Approach (2001)

Building on the receding horizon approach by Hernandez-Lerma andLasserre in solving Markov decision processes (MDPs),this paper first analyzes the performance of the (approximate) receding horizon...

Estimation of Hidden Markov Models (2001)

We consider a receding horizon approach as an approximate solution totwo-person zero-sum Markov games with infinite horizon discounted costand average cost criteria. We first present error bounds...

Ramezani, Vahid Reza, Marcus, Steven I.

A risk-sensitive generalization of the Maximum A Posterior Probability (MAP) estimationfor partially observed Markov chains is presented.Using a change of measure technique,a cascade filtering scheme...

Approximate Receding Horizon Approach for Markov Decision Processes: Average Reward Case (2001)

Markov Games: Receding Horizon Approach (2001)

Building on the receding horizon approach by Hernandez-Lerma andLasserre in solving Markov decision processes (MDPs),this paper first analyzes the performance of the (approximate) receding horizon...

Two-Timescale Simultaneous Perturbation Stochastic Approximation Using Deterministic Perturbation Sequences (2001)

We consider a receding horizon approach as an approximate solution totwo-person zero-sum Markov games with infinite horizon discounted costand average cost criteria. We first present error bounds...

Shalabh Bhatnagar \lambda, Michael C. Fu, Steven I. Marcus

1

Optimal Structured Feedback Policies for ABR Flow Control using Two-Timescale SPSA \Lambda (2001)

Shalabh Bhatnagar Y, Michael C. Fu, Steven I. Marcus, Pedram Jaefari, M. Fard

Optimal structured feedback policies for ABR flow control using two-timescale SPSA (2001)

Shalabh Bhatnagar, Steven I. Marcus, Pedram Jaefari, M. Fard

Optimal structured feedback control policies for rate-based flow control of available bit rate (ABR) service in asynchronous transfer mode (ATM) networks are obtained in the presence of information...

Randomized Difference Two-Timescale Simultaneous Perturbation Stochastic Approximation Algorithms for Simulation Optimization of Hidden Markov Models (2000)

Bhatnagar, Shalabh, Fu, Michael C., Marcus, Steven I., Bhatnagar, Shashank

We proposetwo finite difference two-timescale simultaneous perturbationstochastic approximation (SPSA)algorithmsfor simulation optimization ofhidden Markov models. Stability and convergence of both...

Stability of Wireless Networks for Mode S Radar (2000)

Chawla, Jay P., Marcus, Steven I., Shayman, Mark A.

Stability issues in a connectionless, one-hop queueing system featuringservers with overlapping service regions (e.g. a Mode Select (Mode S) Radarcommunications network or part of an Aeronautical...

Simulation-Based Approach for Semiconductor Fab-Level Decision Making - Implementation Issues (2000)

Approximate Policy Iteration for Semiconductor Fab-Level Decision Making - a Case Study (2000)

In this paper, we discuss implementation issues of applying a simulation-based approach to asemiconductor fab-level decision making problem. The fab-level decision making problem isformulated as a...

He, Ying, Bhatnagar, Shalabh, Fu, Michael C., Marcus, Steven I.

In this paper, we propose an approximate policy iteration (API) algorithm for asemiconductor fab-level decision making problem. This problem is formulated as adiscounted cost Markov Decision Process...

Randomized Difference Two-Timescale Simultaneous Perturbation Stochastic Approximation Algorithms for Simulation Optimization of Hidden Markov Models (2000)

Bhatnagar, Shalabh, Fu, Michael C., Marcus, Steven I., Bhatnagar, Shashank

We proposetwo finite difference two-timescale simultaneous perturbationstochastic approximation (SPSA)algorithmsfor simulation optimization ofhidden Markov models. Stability and convergence of both...

Stability of Wireless Networks for Mode S Radar (2000)

Chawla, Jay P., Marcus, Steven I., Shayman, Mark A.

Stability issues in a connectionless, one-hop queueing system featuringservers with overlapping service regions (e.g. a Mode Select (Mode S) Radarcommunications network or part of an Aeronautical...

Simulation-Based Approach for Semiconductor Fab-Level Decision Making - Implementation Issues (2000)

Approximate Policy Iteration for Semiconductor Fab-Level Decision Making - a Case Study (2000)

In this paper, we discuss implementation issues of applying a simulation-based approach to asemiconductor fab-level decision making problem. The fab-level decision making problem isformulated as a...

He, Ying, Bhatnagar, Shalabh, Fu, Michael C., Marcus, Steven I.

In this paper, we propose an approximate policy iteration (API) algorithm for asemiconductor fab-level decision making problem. This problem is formulated as adiscounted cost Markov Decision Process...

Randomized difference two-timescale simultaneous perturbation stochastic approximation algorithms for simulation optimization of hidden Markov models (2000)

Shalabh Bhatnagar, Michael C. Fu, Steven I. Marcus, Shashank Bhatnagar

We propose two finite difference two-timescale simultaneous perturbation stochastic approximation (SPSA) algorithms for simulation optimization of hidden Markov models. Stability and convergence of...

A Simulation-Based Policy Iteration Algorithm for Average Cost Unichain Markov Decision Processes (2000)

Ying He, Michael C. Fu, Steven I. Marcus

In this paper, we propose a simulation-based policy iteration algorithm on Markov decision process (MDP) problems with average cost criterion under the unichain assumption, which is a weaker...

Mixed Risk-Neutral/Minimax Control of Discrete-Time, Finite-State Markov Decision Processes (2000)

Mixed Risk-Neutral/Minimax Control of Discrete-Time, Finite-State Markov Decision Processes (2000)

This paper addresses the control design problem for discrete-time, finite-state Markov Decision Processes (MDPs), when both risk-neutral and minimax objectives are of interest. We introduce the mixed...

Monotone Optimal Policies for a Transient Queueing Staffing Problem (2000)

This paper addresses the control design problem for discrete-time, finite-state Markov Decision Processes (MDPs), when both risk-neutral and minimax objectives are of interest. We introduce the mixed...

Michael Fu, Steven I. Marcus, I-jeng Wang

We consider the problem of determining the optimal policy for staffing a queueing system over multiple periods, using a model that takes into account transient queueing effects. Formulating the...

Monotone Optimal Policies for a Transient Queueing Staffing Problem (2000)

Michael C. Fu, Steven I. Marcus, I-jeng Wang

We consider the problem of determining the optimal policy for staffing a queueing system over multiple periods, using a model that takes into account transient queueing effects. Formulating the...

Optimal Multilevel Feedback Policies for ABR Flow Control using Two Timescale SPSA (1999)

Bhatnagar, Shalabh, Fu, Michael C., Marcus, Steven I.

Optimal multilevel control policies for rate based flow control in available bit rate (ABR) service in asynchronous transfer mode (ATM) networks are obtained in the presence of information and...

Simulation-Based Algorithms for Average Cost Markov Decision Processes (1999)

Optimal Multilevel Feedback Policies for ABR Flow Control using Two Timescale SPSA (1999)

In this paper, we give a summary of recent development of simulation-based algorithmsfor average cost MDP problems, which are different from those for discounted cost problems or shortest...

Bhatnagar, Shalabh, Fu, Michael C., Marcus, Steven I.

Optimal multilevel feedback control policies for rate based flow controlin available bit rate (ABR) service in asynchronous transfer mode (ATM)networks are obtained in the presence of information and...

Optimal Multilevel Feedback Policies for ABR Flow Control using Two Timescale SPSA (1999)

Bhatnagar, Shalabh, Fu, Michael C., Marcus, Steven I.

Optimal multilevel control policies for rate based flow control in available bit rate (ABR) service in asynchronous transfer mode (ATM) networks are obtained in the presence of information and...

Simulation-Based Algorithms for Average Cost Markov Decision Processes (1999)

Optimal Multilevel Feedback Policies for ABR Flow Control using Two Timescale SPSA (1999)

In this paper, we give a summary of recent development of simulation-based algorithmsfor average cost MDP problems, which are different from those for discounted cost problems or shortest...

Bhatnagar, Shalabh, Fu, Michael C., Marcus, Steven I.

Optimal multilevel feedback control policies for rate based flow controlin available bit rate (ABR) service in asynchronous transfer mode (ATM)networks are obtained in the presence of information and...

Risk-Sensitive and Minimax Control of Discrete-Time, Finite-State Markov Decision Processes (1999)

Stefano Coraluppi And, S. I. Marcus, Steven I. Marcus

This paper analyzes a connection between risk-sensitive and minimax criteria for discrete-time, #nite-states Markov Decision Processes #MDPs#. We synthesize optimal policies with respect to both...

A Markov Decision Process Model for Capacity Expansion and Allocation (1999)

Shalabh Bhatnagar, Emmanuel Fernandez-Gaucherand, Michael C. Fu, Ying He, Steven I. Marcus

We present a finite-horizon Markov decision process (MDP) model for providing decision support in semiconductor manufacturing on such critical operational issues as when to add additional capacity...

Change Detection In Markov-Modulated Time Series (1999)

Subhrakanti Dey, Steven I. Marcus

In this paper, we address the problem of online change detection of Markov-modulated time series models. For simplicity, we look at Auto-regressive time-series models the parameters of which are...

ISR develops, applies and teaches advanced methodologies of design and analysis to solve complex, hierarchical, (1999)

Heterogeneous And Dynamic, Shalabh Bhatnagar, Shalabh Bhatnagar, Michael C. Fu, Michael C. Fu, Steven I. Marcus, ...

Optimal multilevel feedback control policies for rate based flow control in available bit rate (ABR) service in asynchronous transfer mode (ATM) networks are obtained in the presence of information...

Optimal Multilevel Feedback Policies for ABR Flow Control using Two Timescale SPSA (1999)

Risk-Sensitive and Minimax Control of Discrete-Time, Finite-State Markov Decision Processes (1999)

Optimal multilevel feedback control policies for rate based flow control in available bit rate (ABR) service in asynchronous transfer mode (ATM) networks are obtained in the presence of information...

Stefano Coraluppi And, Steven I. Marcus

This paper analyzes a connection between risk-sensitive and minimax criteria for discrete-time, finite-states Markov Decision Processes (MDPs). We synthesize optimal policies with respect to both...

Change Detection In Markov-Modulated Time Series (1999)

Subhrakanti Dey Dept, Subhrakanti Dey, Steven I. Marcus

In this paper, we address the problem of online change detection of Markov-modulated time series models. For simplicity, we look at Auto-regressive time-series models the parameters of which are...

Harnack's Inequality For Cooperative Weakly Coupled Elliptic Systems (1999)

Aristotle Arapostathis, Mrinal K. Ghosh, Steven I. Marcus

this paper, we obtain analogues of Harnack's inequality for L-harmonic functions of operators in the class L(#, d, n). We use the technique introduced by Krylov for estimating the oscillation of...

Risk-Sensitive and Minimax Control of Discrete-Time, Finite-State Markov Decision Processes (1999)

Harnack's inequality for cooperative weakly coupled elliptic systems (1999)

This paper analyzes a connection between risk-sensitive and minimax criteria for discrete-time, finite-states Markov Decision Processes (MDPs). We synthesize optimal policies with respect to both...

Arapostathis, Ari, Ghosh, Mrinal K, Marcus, Steven I

Reduced-Order Estimators and Their Application to Aircraft Navigation. (1998)

Center, Julian L., D'Appolito, Joseph A., Marcus, Steven I.

Minimum-variance reduced-order estimators are derived and analyzed. Although the computations required to determine the optimum gains and dynamics for these optimal reduced-order filters and...

Estimation for Certain Nonlinear Stochastic Systems. (1998)

Adaptive Control of Nonlinear and Stochastic Systems. (1998)

Recently a great deal of attention has gone into the study of classes of deterministic nonlinear systems described either by bilinear differential state equations or Volterra series input-output...

Marcus, Steven I., Arapostathis, Aristotle

Significant progress was made in a number of aspects of nonlinear and stochastic systems. An important problem in the adaptive estimation of a finite state Markov chain was solved, and significant...

Adaptive Control of Nonlinear and Stochastic Systems. (1998)

Marcus, Steven I., Arapostathis, Aristotle

Significant progress was made in a number of aspects of nonlinear and stochastic systems. An important problem in the adaptive control of a finite state Markov chain was solved, and significant...

Integrated Simulation-Based Methodologies for Planning and Estimation (1998)

Marcus, Steven I., Fu, Michael C., Willsky, Alan S.

Significant progress was made in a number of proposed research areas. The first major task in the proposal involved incorporating simulation-based optimization (and, in particular, ordinal...

Approximate Receding Horizon Approach for Markov Decision Processes: Average Award Case (1998)

Chang, Hyeong S., Marcus, Steven I.

The authors consider an approximation scheme for solving Markov Decision Processes (MDPs) with countable state space, finite action space, and bounded rewards that uses an approximate solution of a...

Markov Games: Receding Horizon Approach (1998)

Chang, Hyeong S., Marcus, Steven I.

The authors consider a receding horizon approach as an approximate solution to two-person, zero-sum Markov games with infinite horizon discounted cost and average cost criteria. They first present...

An Adaptive Sampling Algorithm for Solving Markov Decision Processes (1998)

Chang, Hyeong S., Fu, Michael C., Marcus, Steven I.

Based on recent results for multi-armed bandit problems, the authors propose an adaptive sampling algorithm that approximates the optimal value of a finite horizon Markov decision process (MDP) with...

Multi-time Scale Markov Decision Processes (1998)

Chang, Hyeong S., Fard, Pedram, Marcus, Steven I., Shayman, Mark

This paper proposes a simple analytical model called M time-scale Markov Decision Process (MMDP) for hierarchically structured sequential decision-making processes, where decisions at each level in...

Risk-Sensitive Probability for Markov Chains (1998)

Ramezani, Vahid R., Marcus, Steven I.

The probability distribution of a Markov chain is viewed as the information state of an additive optimization problem. This optimization problem is then generalized to a product form whose...

Harnack's Inequality for Cooperative Weakly Coupled Elliptic Systems (1998)

Arapostathis, Aristotle, Ghosh, Mrinal K., Marcus, Steven I.

The authors consider cooperative, uniformly elliptic systems, with bounded coefficients and coupling in the zeroth-order terms. They establish two analogs of Harnack's inequality for this class of...

A Framework for Mixed Estimation of Hidden Markov Models (1998)

Dey, Subhrakanti, Marcus, Steven I.

In this paper, we present a framework for a mixed estimation scheme for hidden Markov models (HMM). A robust estimation scheme is first presented using the minimax method that minimizes a worst case...

Stability of Wireless Networks for Mode S Radar (1998)

Chawla, Jay P., Marcus, Steven I., Shayman, Mark A.

Stability issues in a connectionless, one-hop queueing system featuring servers with overlapping service regions (e.g., a Mode Select (Mode S) radar communications network or part of an Aeronautical...

A NEW APPROACH TO PRICING AMERICAN-STYLE DERIVATIVES (1998)

Laprise, Scott B., Fu, Michael C., Marcus, Steven I., Lim, Andrew E.B.

This paper presents a new approach to pricing American-style derivatives. By approximating the value function with a piecewise linear interpolation function, the option holder s continuation value...

Risk-Sensitive and Minimax Control of Discrete-Time, Finite-State Markov Decision Processes (1998)

Coraluppi, Stephano P., Marcus, Steven I.

This paper analyzes a connection between risk-sensitive and minimaxcriteria for discrete-time, finite-states Markov Decision Processes(MDPs). We synthesize optimal policies with respect to both...

Risk-Sensitive, Minimax, and Mixed Risk-Neutral/Minimax Control of Markov Decision Processes (1998)

Stefano Coraluppi, Steven I. Marcus, Steven I. Marcus

: This paper analyzes a connection between risk-sensitive and minimax criteria for discrete-time, #nite-state Markov Decision Processes #MDPs#. We synthesize optimal policies with respect to both...

Ergodic control of switching diffusions (1997)

Ghosh, Mrinal K, Arapostathis, Aristotle, Marcus, Steven I

We study the ergodic control problem of switching diusions representing a typical hybrid system that arises in numerous applications such as fault-tolerant control systems, flexible manufacturing...

Stochastic Adaptive Estimation and Control. (1997)

Marcus, Steven I.

Significant progress was made in a number of aspects of stochastic and discrete event systems. A controlled switching diffusion model was developed to study systems with multiple modes or failure...

Ergodic control of switching diffusions (1997)

Ghosh, Mrinal K, Arapostathis, Aristotle, Marcus, Steven I

We study the ergodic control problem of switching diusions representing a typical hybrid system that arises in numerous applications such as fault-tolerant control systems, flexible manufacturing...

Monotone Optimal Policies for a Transient Queueing Staffing Problem (1997)

Fu, Michael C., Marcus, Steven I., Wang, I-Jeng

We consider the problem of determining the optimal policy for staffing a queueing system over multiple periods, using a model that takes into account transient queueing effects. Formulating the...

Existence of Risk Sensitive Optimal Stationary Policies for Controlled Markov Processes (1997)

Hernandez-Hernandez, Daniel, Marcus, Steven I.

In this paper we are concerned with the existence of optimal stationary policies for infinite horizon risk sensitive Markov control processes with denumerable state space, unbounded cost function,...

Monotone Optimal Policies for a Transient Queueing Staffing Problem (1997)

Fu, Michael C., Marcus, Steven I., Wang, I-Jeng

We consider the problem of determining the optimal policy for staffing a queueing system over multiple periods, using a model that takes into account transient queueing effects. Formulating the...

Monotone Optimal Policies for a . . . (1997)

M. C. Fu, S. I. Marcus, I-j. Wang, Michael C. Fu, Steven I. Marcus, I-jeng Wang

We consider the problem of determining the optimal policy for staffing a queueing system over multiple periods, using a model that takes into account transient queueing effects. Formulating the...

A Discrete Event Systems Approach for Protocol Conversion (1997)

Ratnesh Kumar Sudhir, Sudhir Nelvagal, Steven I. Marcus

A protocol mismatch occurs when heterogeneous networks try to communicate with each other. Such mismatches are inevitable due to the proliferation of a multitude of networking architectures,...

Probabilistic Language Formalism for Stochastic Discrete Event Systems (1997)

Vijay Garg, Ratnesh Kumar, Steven I. Marcus

The formalism of probabilistic languages has been introduced for modeling the qualitative behavior of stochastic discrete event systems. A probabilistic language is a unit interval valued map over...

Reinforcement Learning For MDPs Using Temporal Difference Schemes (1997)

Abraham Thomas, Steven I. Marcus

In this paper we propose a reinforcement learning scheme for finding optimal and sub-optimal policies for the finite state Markov Decision Problem (MDP) with the infinite horizon discounted cost...

Mixed Risk-Neutral/Minimax Control of Markov Decision Processes (1997)

Stefano Coraluppi And, Steven I. Marcus

This paper introduces a formulation of the mixed risk-neutral/minimax control problem for Markov Decision Processes (MDPs). Drawing on results from risk-neutral control and minimax control, we derive...

Reinforcement Learning in MDP Framework Using Temporal Difference Schemes (1997)

Abraham Thomas, Steven I. Marcus

In this paper we propose a reinforcement learning scheme for finding optimal and sub-optimal policies for the finite state Markov Decision Problem (MDP) with the infinite horizon discounted cost...

A Discrete Event Systems Approach for Protocol Conversion (1997)

Ratnesh Kumar, Sudhir Nelvagal, Steven I. Marcus

A protocol mismatch occurs when heterogeneous networks try to communicate with each other. Such mismatches are inevitable due to the proliferation of a multitude of networking architectures,...

Mixed Risk-Neutral/Minimax Control of Markov Decision Processes (1997)

Existence of Risk Sensitive Optimal Stationary Policies for Controlled Markov Processes (1997)

This paper introduces a formulation of the mixed risk-neutral/minimax control problem for Markov Decision Processes (MDPs). Drawing on results from risk-neutral control and minimax control, we derive...

Daniel Hernández-Hernández, Steven I. Marcus

In this paper we are concerned with the existence of optimal stationary policies for infinite horizon risk sensitive Markov control processes with denumerable state space, unbounded cost function,...

Analysis of a Risk Sensitive Control Problem for Hidden Markov Chains (1997)

Daniel Hernández-Hernández, Steven I. Marcus, Pedram J. Fard

Introduction Recently considerable attention has been given to the study of risksensitive stochasic control problems [1]-[13]. Risk-sensitive control is a generalization of the risk-neutral approach,...

A Discrete Event Systems Approach for Protocol Conversion (1997)

Ratnesh Kumar, Sudhir Nelvagal, Steven I. Marcus

A protocol mismatch occurs when heterogeneous networks try to communicate with each other. Such mismatches are inevitable due to the proliferation of a multitude of networking architectures,...

Existence of Risk Sensitive Optimal Stationary Policies for Controlled Markov Processes (1997)

Daniel Hern'andez-Hern'andez, Steven I. Marcus

In this paper we are concerned with the existence of optimal stationary policies for infinite horizon risk sensitive Markov control processes with denumerable state space, unbounded cost function,...

Mixed Risk-Neutral/Minimax Control of Markov Decision Processes (1997)

Risk Sensitive Control of Markov Processes in Countable State Space (1996)

This paper addresses the control design problem for discrete-time, finitestate Markov Decision Processes (MDPs), when both risk-neutral and minimax objectives are of interest. We introduce the mixed...

Hernandez-Hernandez, Daniel, Marcus, Steven I.

In this paper we consider infinite horizon risk-sensitive control of Markov processes with discrete time and denumerable state space. This problem is solved proving, under suitable conditions, that...

Probabilistic Language Framework for Stochastic Discrete Event Systems (1996)

Garg, Vijay K., Kumar, Ratnesh, Marcus, Steven I.

We introduce the notion of probabilistic languages to describe the qualitative behavior of stochastic discrete event systems. Regular language operators such as choice, concatenation, and...

A Note on an LQG Regulator with Markovian Switching and Pathwise Average Cost (1995)

Ghosh, Mrinal K, Arapostathis, Aristotle, Marcus, Steven I

We study a linear system with a Markovian switching parameter perturbed by white noise. The cost function is quadratic. Under certain conditions, we find a linear feedback control which is almost...

A Note on an LQG Regulator with Markovian Switching and Pathwise Average Cost (1995)

Ghosh, Mrinal K, Arapostathis, Aristotle, Marcus, Steven I

We study a linear system with a Markovian switching parameter perturbed by white noise. The cost function is quadratic. Under certain conditions, we find a linear feedback control which is almost...

Extension Based Limited Lookahead Supervision of Discrete Event Systems (1995)

Kumar, Ratnesh, Cheung, Hok M., Marcus, Steven I.

Supervisory control of discrete event systems using limited lookahead has been studied by Chung-Lafortune-Lin, where control is computed by truncating the plant behavior up to the limited lookahead...

Predicates and Predicate Transformers for Supervisory Control of Discrete Event Dynamical Systems (1995)

Ratnesh Kumar, Vijay Garg, Steven I. Marcus

Most discrete event system models are based on defining the alphabet set or the set of events as a fundamental concept. In this paper, we take an alternative view of treating the state space as the...

Finite Buffer Realization of Input-Output Discrete Event Systems (1995)

Finite Buffer Realization of Input-Output Discrete Event Systems (1995)

Many discrete event systems (DESs) such as manufacturing systems, database management systems, communication networks, traffic systems, etc., can be modeled as input-output discrete event systems...

Optimal Structured Feedback Policies for ABR Flow Control Using Two Timescale SPSA (1994)

Many discrete event systems (DESs) such as manufacturing systems, data base management systems, communication networks, traffic systems, etc., can be modeled as input-output discrete event systems...

Shalabh Bhatnagar, Michael C. Fu, Steven I. Marcus, Pedram J. Fard

Abstract—Optimal structured feedback control policies for rate-based flow control of available bit rate service in asynchronous transfer mode networks are obtained in the presence of information...

Optimal Control of Switching Diffusions with Application to Flexible Manufacturing Systems (1993)

Ghosh, Mrinal K, Arapostathis, Aristotle, Marcus, Steven I

A controlled switching diffusion model is developed to study the hierarchical control of flexible manufacturing systems. The existence of a homogeneous Markov nonrandomized optimal policy is...

Optimal Control of Switching Diffusions with Application to Flexible Manufacturing Systems (1993)

Ghosh, Mrinal K, Arapostathis, Aristotle, Marcus, Steven I

A controlled switching diffusion model is developed to study the hierarchical control of flexible manufacturing systems. The existence of a homogeneous Markov nonrandomized optimal policy is...

Controlled Markov Processes on the Infinite Planning Horizon: Weighted and, Overtaking Cost Criteria (1993)

Fernandez-Gaucherand, Emmanuel, Ghosh, Mrinal K., Marcus, Steven I.

Stochastic control problems for controlled Markov processes models with an infinite planning horizon are considered, under some non-standard cost criteria. The classical discounted and average cost...

Language Stability and Stabilizability of Discrete Event Dynamical Systems (1993)

Ratnesh Kumar, Vijay Garg, Steven I. Marcus

This paper studies the stability and stabilizability of Discrete Event Dynamical Systems (DEDS's) modeled by state machines. We define stability and stabilizability in terms of the behavior of...

Optimal Control of A Hybrid System With Pathwise Average Cost (1992)

Ghosh, Mrinal K, Arapostathis, Aristotle, Marcus, Steven I

We study the ergodic control problem of switching diffusions representing a typical hybrid system that arises in numerous applications such as fault tolerant control systems, flexible manufacturing...

Optimal Control of A Hybrid System With Pathwise Average Cost (1992)

Ghosh, Mrinal K, Arapostathis, Aristotle, Marcus, Steven I

We study the ergodic control problem of switching diffusions representing a typical hybrid system that arises in numerous applications such as fault tolerant control systems, flexible manufacturing...

On Supervisory Control of Sequential Behaviors (1992)