iCreatAI News
Skip to content
Google Developers Blog·· 2 days agoAI score58

Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference Mismatch

Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference Mismatch

AI summary

Google describes a Zero train–inference mismatch (Zero-TIM) implementation in Tunix that makes the sampler and trainer assign bitwise-identical logprobs to the same action and token history, verified across 64 TPU v5p chips (DP8×TP8). The approach uses batch-invariant Pallas kernels, explicit cross-shard reduction ordering, a standardized logprob scoring tail, and custom differentiation rules for serving kernels. In a 200-step FrozenLake test with Qwen3-8B, Zero-TIM recorded zero mean absolute sampler–trainer logprob difference, while standard training and token-level truncated importance sampling retained nonzero differences.

Source: Google Developers Blog · developers.googleblog.com