文章目录

Bellman equationDefinition

Bellman equation

Bellman expectation equation 是强化学习中非常基础而且重要的概念，但是有些细节却不好理解，尤其是关于${\mathbb{E}}_{\pi }$ (关于 policy $\pi$ 的期望）的部分。在参考了Understanding RL: The Bellman Equations 和 Derivation of Bellman’s Equation 这两篇文章中的推导内容之后，特地将 Bellman 公式的推理过程整理在这里。

Definition

Given a finite set of states ($S$) and actions ($A$), the state transition probability is $$p(s^{\prime }|s,a)=Pr(S_{t+1}=s^{\prime }|S_t=s,A_t=a)$$ and the reward is $$R(r|s,a,s^{\prime })=Pr(R_{t+1}=r|S_t=s,A_t=a,S_{t+1}=s^{\prime })$$ Notice the reward is actually a distribtion other than a determined value, this is root of some misunderstanding, because some articles only write the expectation.

Here we could put state transition and reward distribution as a single probability $$p(s^{\prime },r|s,a)=Pr(S_{t+1}=s^{\prime },R_{t+1}=r|S_t=s,A_t=a)$$

Given a policy is a mapping from $S$ to $A$ like $\pi (a|s)$.

The value function $$V_{\pi }(s)={\mathbb{E}}_{\pi }\{\sum _{k=1}^{\inf }{\gamma }^k{R}_{t+k+1}|S_t=s\}$$ and action value function as $$q_{\pi }(s,a)={\mathbb{E}}_{\pi }\{\sum _{k=1}^{\inf }{\gamma }^k{R}_{t+k+1}|S_t=s,A_t=a\}$$

According to the well known law of total expectation and the state transition diagram

$$q_{\pi }(s,a)=\sum _{s^{\prime },r}[r+\gamma V_{\pi }(s^{\prime })]p(s^{\prime },r|s,a)$$

$$\begin{gathered} V_{\pi }(s)=\sum _a\pi (a|s)\sum _{s^{\prime },r}[r+\gamma V_{\pi }(s^{\prime })]p(s^{\prime },r|s,a) \\ =\sum _a\pi (a|s)q_{\pi }(s,a) \end{gathered}$$