v1.0.0 CleanRL Release 🎉2022-11-14
🎉 We are thrilled to announce the v1.0.0 CleanRL Release. Along with our CleanRL paper's recent publication in Journal of Machine Learning Research, our v1.0.0 release includes reworked documentation, new algorithm variants, support for google's new ML framework JAX, hyperparameter tuning utilities, and more. CleanRL has come a long way making high-quality deep reinforcement learning implementations easy to understand and reproducible. This release is a major milestone for the project and we are excited to share it with you. Over 90 PRs were merged to make this release possible. We would like to thank all the contributors who made this release possible.
Reworked documentation
One of the biggest change of the v1 release is the added documentation at docs.cleanrl.dev. Having great documentation is important for building a reliable and reproducible project. We have reworked the documentation to make it easier to understand and use. For each implemented algorithm, we have documented as much as we can to promote transparency:
* Short description of the algorithm and references
* A list of implemented variant
* The usage information
* The explanation of the logged metrics
* The documentation of implementation details
* Experimental results
Here is a list of the algorithm variants and their documentation:
| Algorithm | Variants Implemented |
| ----------- | ----------- |
| ✅ Proximal Policy Gradient (PPO) | ppo.py, docs |
| | ppo_atari.py, docs
| | ppocontinuousaction.py, docs
| | ppoatarilstm.py, docs
| | ppoatarienvpool.py, docs
| | ppoatarienvpoolxlajax.py, docs
| | ppo_procgen.py, docs
| | ppoatarimultigpu.py, docs
| | ppopettingzooma_atari.py, docs
| | ppocontinuousaction_isaacgym.py, docs
| ✅ Deep Q-Learning (DQN) | dqn.py, docs |
| | dqn_atari.py, docs |
| | dqn_jax.py, docs |
| | dqnatarijax.py, docs |
| ✅ Categorical DQN (C51) | c51.py, docs |
| | c51_atari.py, docs |
| ✅ Soft Actor-Critic (SAC) | saccontinuousaction.py, docs |
| ✅ Deep Deterministic Policy Gradient (DDPG) | ddpgcontinuousaction.py, docs |
| | ddpgcontinuousaction_jax.py, docs
| ✅ Twin Delayed Deep Deterministic Policy Gradient (TD3) | td3continuousaction.py, docs |
| | td3continuousaction_jax.py, docs |
| ✅ Phasic Policy Gradient (PPG) | ppg_procgen.py, docs |
| ✅ Random Network Distillation (RND) | pporndenvpool.py, docs |
We also improved the contribution guide to make it easier for new contributors to get started. We are still working on improving the documentation. If you have any suggestions, please let us know in the GitHub Issues.
New algorithm variants, support for JAX
We now support JAX-based learning algorithm variants, which are usually faster than the torch equivalent! Here are the docs of the new JAX-based DQN, TD3, and DDPG implementations:
dqnatari_jax.py @kinalmehta in vwxyzjn/cleanrl#222
about 25% faster than dqnatari.py.
td3continuousactionjax.py by @joaogui1 in vwxyzjn/cleanrl#225
about 2.5-4x faster than td3continuous_action.py.
ddpgcontinuousactionjax.py by @vwxyzjn in vwxyzjn/cleanrl#187
about 2.5-4x faster than ddpgcontinuous_action.py.
[ppoatarienvpoolxla_jax.py](https://docs.cleanrl.dev/
v1.0.0b2 JAX Support and Hyperparameter Tuning2022-10-03
🎉 I am thrilled to announce the v1.0.0b2 CleanRL Beta Release. This new release comes with exciting new features. First, we now support JAX-based learning algorithms, which are usually faster than the torch equivalent! Here are the docs of the new JAX-based DQN, TD3, and DDPG implementations:
dqnatari_jax.py
dqnjax.py
td3continuousactionjax.py
ddpgcontinuousactionjax.py
<img width="775" alt="image" src="https://user-images.githubusercontent.com/5555347/193662442-27e5c5af-8c12-4f1d-b5d6-b69d82704614.png">
Also, we now have preliminary support for hyperparameter tuning via optuna (see docs), which is designed to help researchers to find a single set of hyperparameters that work well with a kind of games. The current API looks like below:
``python
import optuna
from cleanrl_utils.tuner import Tuner
tuner = Tuner(
script="cleanrl/ppo.py",
metric="charts/episodic_return",
metriclastnaveragewindow=50,
direction="maximize",
aggregation_type="average",
target_scores={
"CartPole-v1": [0, 500],
"Acrobot-v1": [-500, 0],
},
params_fn=lambda trial: {
"learning-rate": trial.suggest_loguniform("learning-rate", 0.0003, 0.003),
"num-minibatches": trial.suggest_categorical("num-minibatches", [1, 2, 4]),
"update-epochs": trial.suggest_categorical("update-epochs", [1, 2, 4, 8]),
"num-steps": trial.suggest_categorical("num-steps", [5, 16, 32, 64, 128]),
"vf-coef": trial.suggest_uniform("vf-coef", 0, 5),
"max-grad-norm": trial.suggest_uniform("max-grad-norm", 0, 5),
"total-timesteps": 100000,
"num-envs": 16,
},
pruner=optuna.pruners.MedianPruner(nstartuptrials=5),
sampler=optuna.samplers.TPESampler(),
)
tuner.tune(
num_trials=100,
num_seeds=3,
)
`
Besides, we added support for new algorithms/environments, which are
Isaac Gym support in PPO for GPU accelerated robotics environment. ppocontinuousactionisaacgym.py
Random Network Distillation (RND) for highly exploratory environments: ppornd_envpool.py
I would like to cordially thank the core dev members @dosssman @yooceii @Dipamc @kinalmehta for their efforts in helping maintain the CleanRL repository. I would also like to give a shout-out to our new contributors @cool-RR, @Howuhh, @jseppanen, @joaogui1, @kinalmehta, and @ALPH2H.
New CleanRL Supported Publications
Jiayi Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, Zhongwen Xu, & Shuicheng YAN (2022). EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/forum?id=BubxnHpuMbG
New Features PR
* prototype jax with ddpg by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/187
* Isaac Gym Envs PPO updates by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/233
* JAX TD3 prototype by @joaogui1 in https://github.com/vwxyzjn/cleanrl/pull/225
* prototype jax with dqn by @kinalmehta in https://github.com/vwxyzjn/cleanrl/pull/222
* Poetry 1.2 by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/271
Add rndppo.py documentation and refactor by @yooceii in https://github.com/vwxyzjn/cleanrl/pull/151
* Hyperparameter optimization by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/228
* Update the hyperparameter optimization example script by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/268
Bug Fixes PR
* Td3 ddpg action bound fix by @dosssman in https://github.com/vwxyzjn/cleanrl/pull/211
* added gamma to reward normalization wrappers by @Howuhh in https://github.com/vwxyzjn/cleanrl/pull/209
* Seed envpool environment explicitly by @jseppanen in https://github.com/vwxyzjn/cleanrl/pull/238
* Fix PPO + Isaac Gym Benchmark Script by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/243
* Fix for noise sampling for the TD3 exploration by @dosssman in https://github.com/vwxyzjn/cleanrl/pull/260
Documentation PR
* Add a note on PPG's performance by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/199
* Clarify CleanRL is a non-modular library by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/200
* Fix documentation link by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/213
* JAX + DDPG docs fix by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/229
Fix links in docs for ppocontinuousactionisaacgym.py` by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/242
* Fix docs (badge, TD3 + JAX, and DQN + JAX) by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/246
* Fix typos by @ALPH2H in https://github.com/vwxyzjn/cleanrl/pull/282
* Fix docs links in README.md by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/254
* chore: remove unused parameters in jax implementations by @kinalmehta in https://github.com/vwxyzjn/cleanrl/pull/264
Misc PR
* Show correct exception cause by @cool-RR in https://github.com/vwxyzjn/cleanrl/pull/205
* Remove pettingzoo's pistonball example by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/214
* Leverage CI to speed up poetry lock by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/235
* Ubuntu runner for poetry lock by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/236
* Remove the github pages CI in favor of vercel by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/241
* Clarify LICENSE info by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/253
* Update published paper citation by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/284
* Refactor dqn word choice by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/257
New Contributors
* @cool-RR made their first contribution in https://github.com/vwxyzjn/cleanrl/pull/205
* @Howuhh made their first contribution in https://github.com/vwxyzjn/cleanrl/pull/209
* @jseppanen made their first contribution in https://github.com/vwxyzjn/cleanrl/pull/238
* @joaogui1 made their first contribution in https://github.com/vwxyzjn/cleanrl/pull/225
* @kinalmehta made their first contribution in https://github.com/vwxyzjn/cleanrl/pull/222
* @ALPH2H made their first contribution in https://github.com/vwxyzjn/cleanrl/pull/282
Full Changelog: https://github.com/vwxyzjn/cleanrl/compare/v1.0.0b1...v1.0.0b2
v1.0.0b1 CleanRL Beta Release 🎉2022-06-07
🎉 I am thrilled to announce the v1.0.0b1 CleanRL Beta Release. CleanRL has come a long way making high-quality deep reinforcement learning implementations easy to understand. In this release, we have put a huge effort into revamping our documentation site, making our implementation friendly to use for new users.
I would like to cordially thank the core dev members @dosssman @yooceii @Dipamc77 @bragajj for their efforts in helping maintain the CleanRL repository. I would also like to give a shout-out to our new contributors @ElliotMunro200 and @Dipamc77.
New CleanRL supported publications
* Huang, S., Dossa, R., Raffin, A., Kanervisto, A., & Wang, W. (2022). The 37 Implementation Details of Proximal Policy Optimization, International Conference on Learning Representations 2022 Blog Post Track
* Huang, S., Ontañón, S., (2022). A Closer Look at Invalid Action Masking in Policy Gradient Algorithms, The International FLAIRS Conference Proceedings, 35.
* Schmidt, D., & Schmied, T. (2021). Fast and Data-Efficient Training of Rainbow: an Experimental Study on Atari Deep Reinforcement Learning Workshop at the 35th Conference on Neural Information Processing Systems
New algorithm variants
* Match PPG implementation by @Dipamc77 in https://github.com/vwxyzjn/cleanrl/pull/186
* See the documentation here: https://docs.cleanrl.dev/rl-algorithms/ppg/
* Proper multi-gpu support with PPO by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/178
See the documentation here: https://docs.cleanrl.dev/rl-algorithms/ppo/#ppoatari_multigpupy
* Support Pettingzoo Multi-agent Atari envs with PPO by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/188
See the documentation here: https://docs.cleanrl.dev/rl-algorithms/ppo/#ppopettingzoomaataripy
Refactoring changes
Let ppocontinuous_action.pyonly run 1M steps by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/161
* Change ppo.py's default timesteps by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/164
Enable video recording for ppoprocgen.py by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/166
* Refactor replay based scripts by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/173
Documentation changes
A significant amount of documentation changes (tracked by https://github.com/vwxyzjn/cleanrl/issues/121).
See the overview documentation page here: https://docs.cleanrl.dev/rl-algorithms/overview/
Add ddpgcontinuous_action.py docs by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/137
* Fix DDPG docs' description by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/139
* Fix typo in DDPG docs by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/140
* Fix incorrect links in the DDPG docs by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/142
* DDPG documnetation tweaks; added Q loss equations and light explanation by @dosssman in https://github.com/vwxyzjn/cleanrl/pull/145
Add dqnatari.py documentation by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/124
Add documentation for td3continuous_action.py by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/141
* SAC Documentation - Benchmarks - Minor code tweaks by @dosssman in https://github.com/vwxyzjn/cleanrl/pull/146
Add docs for c51.py and c51atari.py by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/159
* Add docs for dqn.py by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/157
* Address stale documentation by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/169
* Documentation improvement - fix links and mkdocs by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/181
* Improve documentation and contribution guide by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/189
* Fix documentation links in README.md by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/192
* Fix the implemented varaints section in PPO by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/193
Misclanouse changes
* Add Pull Request template by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/122
* Amend license to give proper attribution by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/152
* Introduce better contribution guide by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/154
Fix the default wandb project name in ppoatari_envpool.py by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/160
* Removes unmaintained scripts by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/170
* Add PPO documentation by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/163
* Add docs header by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/174
* Update README.md by @ElliotMunro200 in https://github.com/vwxyzjn/cleanrl/pull/177
Update issuetemplate.md by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/180
* Temporarily Remove PPO-RND by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/190
Utility changes
* Export requirements.txt automatically by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/143
* Auto-upgrade syntax via pyupgrade by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/158
* Introduce benchmark utilities by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/165
New Contributors
* @ElliotMunro200 made their first contribution in https://github.com/vwxyzjn/cleanrl/pull/177
* @Dipamc77 made their first contribution in https://github.com/vwxyzjn/cleanrl/pull/186
Full Changelog: https://github.com/vwxyzjn/cleanrl/compare/v0.6.0...v1.0.0b1
v0.6.0 Major Refactoring2022-03-16
What's Changed
* Update paper citation entry by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/91
* Clean up stale files by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/95
Refactor formats in parseargs by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/78
* Add Gitpod support by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/94
* Reorganize README.md by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/93
* Downgrade setuptools by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/98
* Fix readme links by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/104
* Refactor value based methods by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/102
* Introduce pre-commit pipelines by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/107
* Refactor PPG and PPO for procgen by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/108
* Update documentation on PPG and PPO Procgen by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/112
* Add PPO Atari LSTM example by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/83
* Prototype Envpool Support by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/100
* Fix replay buffer compatibility with mujoco envs by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/113
* Add the isort and black badges by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/119
Refactor parseargs() by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/118
* Add ppo.py documentation by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/120
Replace episodereward with episodic_return by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/125
Refactor ppopettingzoo.py by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/128
* Update gym to 0.23.0 by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/129
* Add SPS and q-values metrics for value-based methods by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/126
* Make seed work again in value methods by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/134
* Remove offline DQN scripts by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/135
Deprecate apexdqn_atari.py by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/136
* Update to gym==0.23.1 by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/138
Full Changelog: https://github.com/vwxyzjn/cleanrl/compare/v0.5.0...v0.6.0
v0.5.02021-11-12
What's Changed
* Use Poetry as the package manager by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/50
* Remove links to deleted code on README algorithms by @FelipeMartins96 in https://github.com/vwxyzjn/cleanrl/pull/54
* Add paper plotting utilities by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/55
* Reorganization of files. by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/56
* Bump Gym's version to 0.21.0 by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/61
* Automatically Download Atari Roms by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/62
* Make Spyder Editor Optional by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/66
* Support Python 3.7.1+ by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/67
ddpgcontinuous: Addded env argument to actor and target actor by @dosssman in https://github.com/vwxyzjn/cleanrl/pull/69
* Add pytest as an optional dependency by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/71
Remove SB3 dependency in ppocontinuous_action.py by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/72
* Add e2e tests by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/70
* Fix #74 SAC consistency in logging and training to match other scripts by @dosssman in https://github.com/vwxyzjn/cleanrl/pull/75
* Add MuJoCo environments support. by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/76
* Only run tests given changes to the cleanrl directory by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/77
* Prototype Documentation Site by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/64
* Cloud Utilities Improvement by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/65
* Import built docker image to local registry by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/80
* Remove docker dummy cache by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/81
* Allow buildx to save to local and push by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/82
* Rollback back PyTorch version for better compatibility by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/84
* Cloud utilities refactor by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/85
* Prepare for 0.5.0 release by @vwxyzjn in https://github.com/vwxyzjn/cleanrl/pull/88
New Contributors
* @FelipeMartins96 made their first contribution in https://github.com/vwxyzjn/cleanrl/pull/54
Full Changelog: https://github.com/vwxyzjn/cleanrl/compare/v0.4.8...v0.5.0
v0.4.82021-05-16
v0.4.72021-05-12
v0.4.62021-05-12
v0.4.52021-04-19
v0.4.4预发布2021-04-16