Abstract:
We introduce a class of models for multidimensional control problems which we call skip-free Markov decision processes on trees. We describe and analyse an algorithm applicable to Markov decision processes of this type that are skip-free in the negative direction. Starting with the finite average cost case, we show that the algorithm combines the advantages of both value iteration and policy iteration -- it is guaranteed to converge to an optimal policy and optimal value function after a finite number of iterations but the computational effort required for each iteration step is comparable with that for value iteration. We show that the algorithm can also be used to solve discounted cost models and continuous time models, and that a suitably modified algorithm can be used to solve communicating models.

Abstract:
Many problems in computing, service, and manufacturing systems can be modeled via infinite repeating Markov chains with an infinite number of levels and a finite number of phases. Many such chains are quasi-birth-death processes (QBDs) with transitions that are skip-free in level, in that one can only transition between consecutive levels, and unidirectional in phase, in that one can only transition from lower-numbered phases to higher-numbered phases. We present a procedure, which we call Clearing Analysis on Phases (CAP), for determining the limiting probabilities of such Markov chains exactly. The CAP method yields the limiting probability of each state in the repeating portion of the chain as a linear combination of scalar bases raised to a power corresponding to the level of the state. The weights in these linear combinations can be determined by solving a finite system of linear equations.

Abstract:
For a skip-free Markov process on non-negative integers with generator matrix Q, we evaluate the joint Laplace transform of the occupation times before hitting the state n (starting at 0). This Laplace transform has a very straightforward and familiar expression. We investigate the properties of this Laplace transform, especially the conditions under which the occupation times form a Markov chain.

Abstract:
A fluctuation theory and, in particular, a theory of scale functions is developed for upwards skip-free L\'evy chains, i.e. for right-continuous random walks embedded into continuous time as compound Poisson processes. This is done by analogy to the spectrally negative class of L\'evy processes -- several results, however, can be made more explicit/exhaustive in our compound Poisson setting. In particular, the scale functions admit a linear recursion, of constant order when the support of the jump measure is bounded, by means of which they can be calculated -- some examples are considered.

Abstract:
Fluid limit techniques have become a central tool to analyze queueing networks over the last decade, with applications to performance analysis, simulation and optimization. In this paper, some of these techniques are extended to a general class of skip-free Markov chains. As in the case of queueing models, a fluid approximation is obtained by scaling time, space and the initial condition by a large constant. The resulting fluid limit is the solution of an ordinary differential equation (ODE) in ``most'' of the state space. Stability and finer ergodic properties for the stochastic model then follow from stability of the set of fluid limits. Moreover, similarly to the queueing context where fluid models are routinely used to design control policies, the structure of the limiting ODE in this general setting provides an understanding of the dynamics of the Markov chain. These results are illustrated through application to Markov chain Monte Carlo methods.

Abstract:
A well-known theorem for an irreducible skip-free chain with absorbing state $d$, under some conditions, is that the hitting (absorbing) time of state $d$ starting from state 0 is distributed as the sum of $d$ independent geometric (or exponential) random variables. The purpose of this paper is to present a direct and simple proof of the theorem in the cases of both discrete and continuous time skip-free Markov chains. Our proof is to calculate directly the generation functions (or Laplace transforms) of hitting times in terms of the iteration method.

Abstract:
An (upward) skip-free Markov chain with the set of nonnegative integers as state space is a chain for which upward jumps may be only of unit size; there is no restriction on downward jumps. In a 1987 paper, Brown and Shao determined, for an irreducible continuous-time skip-free chain and any d, the passage time distribution from state 0 to state d. When the nonzero eigenvalues nu_j of the generator are all real, their result states that the passage time is distributed as the sum of d independent exponential random variables with rates nu_j. We give another proof of their theorem. In the case of birth-and-death chains, our proof leads to an explicit representation of the passage time as a sum of independent exponential random variables. Diaconis and Miclo recently obtained the first such representation, but our construction is much simpler. We obtain similar (and new) results for a fastest strong stationary time T of an ergodic continuous-time skip-free chain with stochastically monotone time-reversal started in state 0, and we also obtain discrete-time analogs of all our results. In the paper's final section we present extensions of our results to more general chains.

Abstract:
Recent research of the author has given an explicit geometric description of free (two-sided) adequate semigroups and monoids, as sets of labelled directed trees under a natural combinatorial multiplication. In this paper we show that there are natural embeddings of each free right adequate and free left adequate semigroup or monoid into the corresponding free adequate semigroup or monoid. The corresponding classes of trees are easily described and the resulting geometric representation of free left adequate and free right adequate semigroups is even easier to understand than that in the two-sided case. We use it to establish some basic structural properties of free left and right adequate semigroups and monoids.

Abstract:
We prove a multidimensional Poisson limit theorem in free probability, and define joint free Poisson distributions in a non-commutative probability space. We define (compound) free Poisson process explicitly, similar to the definitions of (compound) Poisson processes in classical probability. We proved that the sum of finitely many freely independent compound free Poisson processes is a compound free Poisson processes. We give a step by step procedure for constructing a (compound) free Poisson process. A Karhunen-Loeve expansion theorem for centered free Poisson processes is proved. We generalize free Poisson processes to a notion of free Poisson random measures (which is slightly different from the previously defined ones in free probability, but more like an analogue of classical Poisson random measures). Then we develop the integration theory of real-valued functions with respect to a free Poisson random measure, generalizing the classical integration theory to the free probability case. We find that the integral of a function (in certain spaces of functions) with respect to a free Poisson random measure has a compound free Poisson distribution. For centered free Poisson random measures, we can get a simpler and more beautiful integration theory.

Abstract:
We study a family of free stochastic processes whose covariance kernels $K$ may be derived as a transform of a tempered measure $\sigma$. These processes arise, for example, in consideration non-commutative analysis involving free probability. Hence our use of semi-circle distributions, as opposed to Gaussians. In this setting we find an orthonormal bases in the corresponding non-commutative $L^2$ of sample-space. We define a stochastic integral for our family of free processes.