Algorithm parameters: step size $\alpha \in (0,1],ϵ>0$
Initialize $Q(s,a),\mathrm{\forall }s\in S^{+},a\in A(s),$ arbitrarily except that $Q(terminal,\cdot )=0$

Loop for each episode:
$$Initialize $S$
$$Loop for each step of episode:
$$Choose $A$ from $S$ using some policy derived from $Q$ (eg $ϵ$-greedy)
$$Take action $A$, observe $R,S^{\prime }$
$Q(S,A)\leftarrow Q(S,A)+\alpha [R+\gamma \underset{a}{max}(S^{\prime },a)-Q(S,A)]$
$S\leftarrow S^{\prime }$
$$ until $S$ is terminal