Zero Mismatch, Simple Async RL: REINFORCE Zero, DeepSeek-V4.1 and Kimi K3.1
Zero Mismatch, Simple Async RL: REINFORCE Zero, DeepSeek-V4.1 and Kimi K3.1
The post argues that asynchronous RL's rollout-to-learner logprob gap splits into smoothly varying weight lag, which unclipped importance sampling can correct, and fluctuating kernel mismatch, which persists despite IS; aligning kernels first lets REINFORCE Zero train stably for 20,000 updates, learn at policy lag 64, and reach 2.40× the training-step throughput of synchronous training with aligned kernels. It reports zero mismatch (bit-for-bit logprob agreement) for DeepSeek-V4.1-Flash and Kimi K3.1 on day 0, and for DeepSeek-V4-Flash and Kimi K3.1 on AMD MI355X. Code and recipes are in Inferact/zero-mismatch-rl, with the paper at hijkzzz/REINFORCE-Zero and training code in NVIDIA-NeMo/labs-molt.
It offers a concrete diagnosis of why async RL drifts and a minimal recipe, backed by throughput, lag-64 and cross-vendor alignment results.
Source: vLLM Blog · vllm.ai