16Kolmogorov EquationsIn order to get from state i at time O to state j at time t + s, the process must be in somestatek attimet.Byaddingall possible statesk,wegettheChapman-Kolmogorovequation.Pu(t+s) =)Pik(t)Pkj(s)k=0TheequationsmaybewritteninmatrixtermsasP(t + s) = P(t) · P(s)NoticethatP(O)=I istheidentitymatrix,andthatwhent isaninteger,wehavethatP(t)=[P(1)]t.Itcanbeshown thatthisalsoholdswhent isnotan integer.Bytakingthetimederivative,KolmogorovforwardequationisobtainedPy(t) = )akj Pik(t) -α,Pij(t) =akj Pik(t)k=0k=0k±TheKolmogorovforward equationsmaybewritten inmatrixformatasNTNU-TrondheimP(t) ·A= P(t)Norwegian Universityof莎Scienceand Technologywww.ntnu.edu
16 Kolmogorov Equations In order to get from state ݅ at time 0 to state ݆ at time ݐݏ ,the process must be in some state ݇ at time t . By adding all possible states ݇, we get the Chapman‐Kolmogorov equation. ܲ ݐݏ ൌܲሺݐሻܲሺݏሻ ୀ The equations may be written in matrix terms as ሻݏሺ۾ · ሻݐሺ۾ ൌ ݏ ݐ ۾ Notice that ۾ሺ0ሻ ൌ ܫ is the identity matrix, and that when ݐ is an integer, we have that ۾ሺݐሻ ൌ ሾ۾ሺ1ሻሿ௧. It can be shown that this also holds when ݐ is not an integer. By taking the time derivative, Kolmogorov forward equation is obtained ܲ ሶ ܽൌ ݐ ୀ ஷ ܽൌ ݐ ܲߙെ ݐ ܲ ୀ ܲሺݐሻ The Kolmogorov forward equations may be written in matrix format as ۾ൌۯ· ݐ ۾ ሶ ሺݐሻ
17StateequationWhenweknowthestateattimeO,wemaysimplifythenotationbywritingPui(t)asP,(t)P,(t) =akjPk (t)k=0Inmatrixterms,theKolmogorovequationcanbewrittenaor/aooa01a1oaiiair[Po(t),., P(t)][Po(t),..,Pr(t)]：：aroariarrorinamorecompactformasP(t) ·A = P(t)NTNU-Trondheim莎Norwegian University ofScience and Technologywww.ntnu.edu
17 State equation When we know the state at time 0, we may simplify the notation by writing ܲሺݐሻ as ܲ(ݐ.( ܲ ሶ ܲܽൌ ݐ ୀ ሺݐሻ In matrix terms, the Kolmogorov equation can be written · ݐ ܲ,., ݐ ܲ ܽ ܽଵ ܽଵ ܽଵଵ ⋯ ܽ ܽଵ ⋮ ⋱⋮ ܽ ܽଵ ⋯ ܽ ൌ ܲ ሶ ܲ,., ݐ ሶ ݐ or in a more compact form as ۾ൌۯ· ݐ ۾ ሶ ሺݐሻ
18SteadystateprobabilitiesSometimes,onlythelong-run(steadystate)probabilitiesareofinterest,thatis,thevaluesofP,(t) when t → o0.Toensurethelimitingstateprobabilitiesexist,wehavetoassumethatAll states inaMarkovprocesscancommunicate.Thismeansthat ifaprocessstartsinstateiat sometimet,ithasa positiveprobabilityof reachingstatejsometime inthefuture.Thismustapplyforanycombinationi,jofstates.The Markov process must bepositive recurrent meaning that if the process starts inaspecified statei,theexpectedtimetoreturntothisstateisfinite,forall statesiNTNU-TrondheimNorwegian University of梦ScienceandTechnologywww.ntnu.edu
18 Steady state probabilities Sometimes, only the long‐run (steady state) probabilities are of interest, that is, the values of ܲሺݐሻ when ݐ.∞→ To ensure the limiting state probabilities exist, we have to assume that • All states in a Markov process can communicate. This means that if a process starts in state ݅ at some time ݐ , it has a positive probability of reaching state ݆ sometime in the future. This must apply for any combination ݅, ݆ of states. • The Markov process must be positive recurrent meaning that if the process starts in a specified state ݅, the expected time to return to this state is finite, for all states ݅