|
| | | | |
| | | |
| Stat |
Members: 3657
Articles: 2'599'751
Articles rated: 2609
08 October 2024 |
|
| | | | |
|
Article overview
| |
|
|
Model Generation with Provable Coverability for Offline Reinforcement Learning | | Chengxing Jia
; Hao Yin
; Chenxiao Gao
; Tian Xu
; Lei Yuan
; Zongzhang Zhang
; Yang Yu
; | | Date: |
1 Jun 2022 | | Abstract: | Model-based offline optimization with dynamics-aware policy provides a new
perspective for policy learning and out-of-distribution generalization, where
the learned policy could adapt to different dynamics enumerated at the training
stage. But due to the limitation under the offline setting, the learned model
could not mimic real dynamics well enough to support reliable
out-of-distribution exploration, which still hinders policy to generalize well.
To narrow the gap, previous works roughly ensemble randomly initialized models
to better approximate the real dynamics. However, such practice is costly and
inefficient, and provides no guarantee on how well the real dynamics could be
approximated by the learned models, which we name coverability in this paper.
We actively address this issue by generating models with provable ability to
cover real dynamics in an efficient and controllable way. To that end, we
design a distance metric for dynamic models based on the occupancy of policies
under the dynamics, and propose an algorithm to generate models optimizing
their coverage for the real dynamics. We give a theoretical analysis on the
model generation process and proves that our algorithm could provide enhanced
coverability. As a downstream task, we train a dynamics-aware policy with minor
or no conservative penalty, and experiments demonstrate that our algorithm
outperforms prior offline methods on existing offline RL benchmarks. We also
discover that policies learned by our method have better zero-shot transfer
performance, implying their better generalization. | | Source: | arXiv, 2206.00316 | | Services: | Forum | Review | PDF | Favorites | |
|
|
No review found.
Did you like this article?
Note: answers to reviews or questions about the article must be posted in the forum section.
Authors are not allowed to review their own article. They can use the forum section.
|
| |
|
|
|
REGISTER