Chapter 2
This chapter is about multiarmed bandits (MAB).
There are two kinds of feedbacks to a RL agent:
Evaluative feedback: how well did it act (a score).
Instructive feedback: best action it could do (a correct answer)
In MAP, the environment is a Markov chain, and thus the past does not influence the future. This makes it easy.
This is called nonassociative learning.
In general,
\[Q_t(a) \approx q_*(a) = \mathbb{E}(R_t | A_t = a)\]
$Q_t(a)$ is the estimate of $q_*(a)$, estimated at time $t$. It is updated as time goes on, hopefully approaching $q_*(a)$ better and better.
The greedy strategy is
\[A_t = \underset{a}{\operatorname{argmax}}Q_t(a)\]
The key is to estimate $Q_t$ accurately. There are several ways.
