REINFORCE Zero asynchronous RL paper released
Understand this story
NVIDIA Research and Inferact have released the paper REINFORCE Zero, which proposes first aligning the kernels on the rollout and training sides, then using uncropped token-level importance sampling to correct weight staleness, enabling asynchronous reinforcement learning to train stably even with a policy lag of 64. According to the paper, with 16 GPUs and 12K updates, its training step throughput is 2.40 times that of synchronous training (68.55 vs. 164.77 hours); the paper says the algorithm can train stably for 20,000 updates. The paper says kernel alignment currently covers DeepSeek-V4.1-Flash (DAPO rollout step gap within 10% of the fast baseline), Kimi K3.1, and DeepSeek-V4-Flash and Kimi K3.1 on AMD MI355X; long-duration asynchronous training on these models remains to be done.
AI overview based on source reports · Updated 1 hr ago
Coverage timeline
Follow the coverage to explore different perspectives.
- vLLM BlogFeaturedZero Mismatch, Simple Async RL: REINFORCE Zero, DeepSeek-V4.1 and Kimi K3.1
The post argues that asynchronous RL's rollout-to-learner logprob gap splits into smoothly varying weight lag, which unclipped importance sampling can correct, and fluctuating kernel mismatch, which persists despite IS; aligning kernels first lets REINFORCE Zero train stably for 20,000 updates, learn at policy lag 64, and reach 2.40× the training-step throughput of synchronous training with aligned kernels. It reports zero mismatch (bit-for-bit logprob agreement) for DeepSeek-V4.1-Flash and Kimi K3.1 on day 0, and for DeepSeek-V4-Flash and Kimi K3.1 on AMD MI355X. Code and recipes are in Inferact/zero-mismatch-rl, with the paper at hijkzzz/REINFORCE-Zero and training code in NVIDIA-NeMo/labs-molt.
Activity for this story
Not enough continuous observations to show a trend yet.