iCreatAI News
Skip to content
Google Developers Blog·· 18 hr agoAI score58

Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference Mismatch

Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference Mismatch

AI summary

Google describes a Zero train–inference mismatch (Zero-TIM) implementation in Tunix that makes sampler decode, serving prefill and trainer forward assign bitwise-identical logprobs to the same sampled action under the same weights, verified across a 64-chip TPU v5p (DP8×TP8) mesh. The recipe combines batch-invariant Pallas kernels with fixed accumulation order, explicit cross-shard reduction ordering, a standardized logprob scoring tail, and custom VJP rules that reuse serving kernels for forward evaluation. In a 200-step FrozenLake test with Qwen3-8B, Zero-TIM recorded zero mean absolute sampler–trainer logprob difference at every logged step, while standard training and token-level truncated importance sampling retained nonzero differences.

Source: Google Developers Blog · developers.googleblog.com