Theoretical Limits of Language Model Alignment

The paper derives information-theoretic limits of KL-regularized alignment, giving a closed-form expression for maximum achievable expected reward gain under a fixed KL budget, governed by a Jeffreys divergence term rather than the √KL used in prior analyses, and reformulates it as a covariance under the base model to yield a practical estimator from base model samples alone. It extends the analysis to proxy rewards, showing the gap between ideal and proxy alignment (reward hacking) grows with reward error magnitude and as the KL penalty factor decreases, and proves reward ensembling mitigates reward hacking. Empirically, on safety and summarization tasks, best-of-N closely approaches the theoretical limit while PPO and GRPO remain substantially suboptimal.