A(high)=search, wait A(low)=search, wait, recharge 1,R wait 1-,-3 search β,R wait 1.0 recharge high search ●wait 1. Rait R search 1-α, R search
Agent State Reward Action Environment S Goal: Learn to choose actions that maximize o+yr,+y-n2+…, where 0≤y<1
Markov Decision processes Assume ● finite set of states s ● set of actions4 at each discrete time agent observes state st E S and chooses action at EA o then receives immediate reward rt and state changes to St+1 Markov assumption: St+1= d(St, at)and t =r(St,at i.e., rt and st+1 depend only on current state and action functions d and r may be nondeterministic - functions o and r not necessarily known to agent
Value function To begin, consider deterministic worlds For each possible policy the agent might adopt we can define an evaluation function over states V(s)=r+7r+1+y2r+2+ r Tt+i where rt, rt+1,.. are generated by following policy T starting at state s Restated, the task is to learn the optimal policy T 丌*≡ argmax v(s),(Vs)
What to learn We might try to have agent learn the evaluation function Vm(which we write as V*) It could then do a lookahead search to choose best action from any state s because T"(s)=argmax[r( s, a)+?V*(8(s, a problem · This works well if agent knows6:S×A→S, andr:S×A4→犹 . But when it doesnt it cant choose actions this way