“Check out MuZero. It learns an embedding of the game's state space potentially allowing AlphaGo like dominance in more ways.”
Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
MuZero:不知规则,靠学到的模型做树搜索,达到超人水平
提出 MuZero,把树搜索与学习得到的环境模型结合:模型只预测与规划最相关的三个量——奖励、动作选择策略和价值函数。它在 57 款 Atari 游戏上刷新了当时的最好成绩,在围棋、国际象棋和将棋上不依赖任何规则知识,也追平了被告知规则的 AlphaZero。
HN 上的人把它当作「学到的世界模型 + 规划」的代表例子,强调它不需要游戏规则也能追平 AlphaZero,并有人推荐其伪代码用来自己实现。
学习路径
读这篇之前,按顺序读书架上的 6 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出从 DQN 与异步 / 分布式深度 RL 入手,再看残差网络(MuZero 的网络骨干)和基于模型的 Atari 方法,最后读 MuZero 如何用学到的模型做树搜索。
- 1
MuZero 在 Atari 上的评测与深度 RL 的价值学习起点,必须先懂。
读Q 学习与经验回放;理解从像素输入学习价值函数的设定。
- 2
A3C 属于 actor-critic 家族,MuZero 的策略与价值预测头沿用这类思想。
读actor-critic 的优势函数和异步更新的基本想法;跳过实现细节。
- 3
- 4
- 5
MuZero 的 Atari 大规模并行训练使用了类似的分布式 actor-learner 思路,并涉及离策略修正。
读actor-learner 分离架构与 V-trace 的作用。
- 6
SimPLe 是 Atari 上基于模型强化学习的代表,对比 MuZero 才看出学习模型但不重建像素的意义。
读视频预测式世界模型的做法与样本效率结论;对照 MuZero 只预测价值、策略、奖励。
- 7Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
目标论文:把学到的隐空间模型与树搜索结合,无需规则即可规划。
读表示、动力学、预测三个函数以及 MCTS 如何使用它们;训练目标的展开(unroll)方式。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“totally! MuZero is my favourite[1]”
“In Go, MuZero slightly exceeded the performance of AlphaZero, despite using less computation per node in the search tree”
“"incorporating a world model" has been going well recently:”
“This line is also very active right now, one example is the recent MuZero.”
“the closest thing I have seen to learned planning is:”
看全部 14 条 →
“this paper includes some pseudocode that implements the algorithm:”
“e.g., MuZero[1] which doesn't even use a model.”
“But it retains MCTS.”
“After AlphaZero, DeepMind went on to develop MuZero ([LINK], paper: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model)”
“This is explained in Appendix A of the paper ("Comparison to AlphaZero"): Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model”
“To a discerning eye this pattern looks like stronger minds implementing higher quality tree search[1], either through implementing higher quality neural search heuristics for tree pruning or via greater search depth.”
“I am personally interested in LLM's, MuZero, Dreamer ([OTHER_PAPER]) and DreamCoder ([OTHER_PAPER])”
“MuZero only masks legal actions at the root of the search tree where the environment can be queried, but does not perform any masking within the search tree.”
- Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model ↗nl · 2019-11-20 · 4 分
- Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model ↗metasj · 2019-11-20 · 161 分
- Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model (2019) ↗tosh · 2023-11-25 · 2 分